Surface the Visible Mirage

Abstract

Embodiments are directed to managing visualizations. A visualization based on data from a data source may be provided. A reputation model based on the visualization may be provided such that the reputation model can detect mirages in the visualization. The reputation model is used to determine reputation results based on the visualization and data from the data source, with each reputation result including a reputation score corresponding to the detection of mirages. The reputation results may be ranked based on the reputation score. A report including a ranked list of the reputation results may be provided.

Description

[Technical field]

[0001] This application is a utility patent application based on previously filed U.S. Provisional Patent Application No. 62 / 902,273 (filed September 18, 2019), the benefit of the filing date of which is hereby claimed under 35 U.S.C. §119(e), and further incorporated by reference in its entirety.

[0002] The present invention relates generally to data visualization, and more particularly, but not exclusively, to automatically identifying potentially misleading visualizations. [Background technology]

[0003] Organizations generate and collect an ever-increasing amount of data. This data may relate to different parts of the organization, such as consumer activities, manufacturing activities, customer service, server logs, and so on. For various reasons, such organizations may find it inconvenient to make effective use of the vast amount of collected data. In some cases, the number of data may make it difficult to make effective use of the collected data to improve business practices. In some cases, organizations use various tools to visualize some or all of the data. Representing this data using visualizations may help organizations improve their understanding of critical business operations and monitor key performance indicators. However, in some cases, even if the visualization appears normal, the visualization may contain mirages that may mislead the creator or viewers of the visualization. In some cases, determining the source or otherwise analyzing the existence, source, or cause of such mirages may require an inordinate understanding of the underlying data used to generate the visualization. Unfortunately, it may be necessary for an organization to instruct a skilled or specialized data analyst to review the visualization and help identify mirages where the visualization may mislead viewers. Also, in some cases, the underlying data may not be accessible to users who could review or verify the visualization, even if the users have the skills and technical background to perform their own Mirage analysis.It is with respect to these and other considerations, therefore, that the present invention has been made. [Brief description of the drawings]

[0004] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following drawings, in which like reference numerals refer to like parts throughout the various drawings unless otherwise specified. For a better understanding of the described innovations, reference is made to the following detailed description of various embodiments, which should be read in conjunction with the accompanying drawings.

[0005] [Figure 1]1 illustrates a system environment in which various embodiments may be implemented. [Diagram 2] 1 illustrates a schematic embodiment of a client computer. [Diagram 3] 1 illustrates a schematic embodiment of a network computer. [Figure 4] 1 illustrates a logical architecture of a system for surfacing a visualization mirage, according to one or more of various embodiments. [Figure 5A] 1 illustrates a representation of a portion of a visualization for surfacing a visualization mirage, according to one or more of the various embodiments. [Figure 5B] 1 illustrates a representation of a portion of a visualization for surfacing a visualization mirage, according to one or more of the various embodiments. [Figure 6A] Here is a portion of a visualization that includes a bar chart comparing house prices. [Figure 6B] Here is a portion of a visualization that includes a bar chart comparing house prices. [Figure 7] 1 illustrates a logical representation of a portion of a rating system according to one or more of various embodiments. [Figure 8] 1 illustrates a logical representation of a portion of a user interface for surfacing a visualization mirage, according to one or more of various embodiments. [Figure 9] 1 depicts a schematic flowchart of a process for surfacing a visualization mirage, according to one or more of various embodiments. [Figure 10] 1 illustrates a flowchart of a process for evaluating visualizations to discover visualization mirages, according to one or more of various embodiments. [Figure 11] 1 illustrates a flowchart of a process for determining a visualization, which may include one or more visualization mirages, according to one or more of various embodiments. [Figure 12] 1 illustrates a flowchart of a process for evaluating a visualization, which may include one or more visualization mirages, according to one or more of various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Various embodiments are described below more fully with reference to the accompanying drawings, which show by way of example specific exemplary embodiments in which the present invention may be practiced, and which form a part of the embodiments. However, the embodiments may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein, but rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the embodiments to those skilled in the art. Among other things, the various embodiments may be methods, systems, media, or devices. Thus, the various embodiments may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Thus, the following detailed description is not to be taken in a limiting sense.

[0007] Throughout the specification and claims, the following terms take on the meanings expressly associated therewith, unless the context clearly indicates otherwise. The phrase "in one embodiment" as used herein does not necessarily refer to the same embodiment, but may. Additionally, the phrase "in another embodiment" as used herein does not necessarily refer to a different embodiment, but may refer to a different embodiment. Thus, as described below, various embodiments may be readily combined without departing from the scope or spirit of the invention.

[0008] Additionally, as used herein, the term "or" is an inclusive "or" operator and is equivalent to the term "and / or" unless context clearly indicates otherwise. The term "based on" is not exclusive and allows for based on additional unrecited factors, unless context clearly indicates otherwise. Additionally, throughout the specification, the meanings of "a," "an," and "the" include plural references. The meaning of "in" includes "in" and "on."

[0009] For example, in embodiments, the following terms are used herein in accordance with the corresponding meanings, unless the context clearly indicates otherwise:

[0010] As used herein, an "engine" refers to logic embodied in hardware or software instructions, which may be written in a programming language such as C, C++, Objective-C, COBOL, Java™, PHP, Perl, JavaScript, Ruby, VBScript, C#, or other Microsoft.NET™ languages. An engine is written in a programming language that is compiled into an executable program or that is interpreted. A software engine may be callable from other engines or itself. An engine as described herein refers to one or more logical modules that may be merged with other engines or applications or may be divided into sub-engines. An engine may be stored in a non-transitory computer-readable medium or computer storage device and stored and executed by one or more general-purpose computers, thus creating a special-purpose computer that is configured to provide the engine.

[0011] As used herein, the term "data source" refers to a database, application, service, file system, etc. that stores or provides information for an organization. Examples of data sources may include RDBMS databases, graph databases, spreadsheets, file systems, document management systems, local or remote data streams, etc. In some cases, data sources are organized around one or more tables or table-like structures. In other cases, data sources are organized as graphs or graph-like structures.

[0012] As used herein, the term "data model" refers to one or more data structures that provide a representation of one or more portions of an underlying data source. In some cases, a data model may provide a view of a data source for a particular application. A data model may be thought of as a view or interface that is based on the data source. In some cases, a data model may map directly to a data source (e.g., a logical pass-through in effect), and in some cases, a data model may be provided by the data source. Data models enable organizations to organize or present information from data sources in ways that are more useful, more meaningful (e.g., easier to reason about), more secure, etc.

[0013] As used herein, the term "data object" refers to one or more data structures that comprise a data model. In some cases, a data object may be considered a portion of a data model. A data object may represent an individual instance of an item or a class or type of item.

[0014] As used herein, the term "panel" refers to an area in a graphical user interface (GUI) that has a defined geometry (e.g., x, y, z axes) within the GUI. A panel may be arranged to display information to a user or to host one or more interactive controls. The geometry or style associated with a panel may be defined using configuration information, including dynamic rules. Also, in some cases, a user may be able to perform actions on one or more panels, such as moving, showing, hiding, resizing, reordering, etc.

[0015] As used herein, a "visualization model" refers to one or more data structures that represent one or more representations of a data model that may be suitable for use in a visualization displayed on one or more hardware displays. A visualization model may define styles or user interface features that may be available to non-author users.

[0016] As used herein, the term "display object" refers to one or more data structures that comprise a visualization model. In some cases, a display object may be considered a portion of a visualization model. A display object may represent an individual instance of an item or an entire class or type of item that may be displayed in a visualization. In some embodiments, a display object may be considered or referred to as a view because it provides a view of a portion of a data model.

[0017] As used herein, the term "visualization mirage" or "mirage" refers to a visually misleading presentation of data in a visualization. Additionally, a visualization mirage may be any visualization in which a cursorial reading of the visualization appears to support a particular message arising from the data, but a closer review of the visualization, the supporting data, or the analytical process invalidates or casts serious doubt on this support.

[0018] As used herein, the term "configuration information" refers to information that may include rule-based policies, pattern matching, scripts (e.g., computer-readable instructions), and the like, which may be provided from a variety of sources including configuration files, databases, user input, built-in defaults, and the like, or combinations thereof.

[0019] The following describes a simplified embodiment of the invention in order to provide a basic understanding of some aspects of the invention. This brief description is not intended to be an extensive overview. It is not intended to identify key or critical elements, to delineate or otherwise narrow the scope. Its purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0020] Briefly, various embodiments are directed to managing visualization of data using one or more processors executing one or more instructions to perform as described herein. In one or more of the various embodiments, one or more visualizations based on data from a data source may be provided.

[0021] In one or more of various embodiments, one or more evaluation models based on the one or more visualizations may be provided, such that the one or more evaluation models may be arranged to detect one or more mirages in the one or more visualizations, and the one or more mirages may be visually misleading presentations of data in the one or more visualizations.

[0022] In one or more of various embodiments, one or more assessment models may be used to determine one or more assessment results based on data from one or more visualizations and data sources, with each assessment result including an assessment score corresponding to the detection of one or more mirages.

[0023] In one or more of various embodiments, determining one or more evaluation results using the one or more evaluation models may include generating one or more test visualizations based on the one or more visualizations, where each test visualization is modified based on the one or more evaluation models; comparing a difference between the one or more test visualizations and the one or more visualizations; and determining a probability score corresponding to detection of the one or more mirages based on the comparison, where the probability value is proportional to the magnitude of the compared differences.

[0024] In one or more of various embodiments, determining one or more evaluation results using the one or more evaluation models may include evaluating data associated with the one or more visualizations to determine one or more of missing or repeated records, spelling errors, drill-down bias, a number of different records per group, or misleading data selection.

[0025] In one or more of the various embodiments, determining one or more evaluation results using one or more evaluation models may include evaluating one or more visualizations to determine one or more types of mirages, including irrational visualizations, overplots, hidden uncertainties, or manipulation of scale.

[0026] In one or more of various embodiments, determining the one or more evaluation results using the one or more evaluation models may further include evaluating data associated with the one or more visualizations that may be included in the data source and excluded from the one or more visualizations, and generating a portion of the one or more evaluation results based on the evaluation.

[0027] In one or more of the various embodiments, one or more evaluation results may be ranked based on the evaluation score.

[0028] In one or more of the various embodiments, a report may be provided that includes a ranked list of the one or more evaluation results. In one or more of the various embodiments, providing the report may include generating one or more other visualizations that may be related to one or more of the one or more evaluation models, the one or more evaluation results, or the one or more mirages.

[0029] Example Operating Environment

[0030] FIG. 1 illustrates components of one embodiment of an environment in which an embodiment of the present invention may be implemented. Not all components may be required to implement the present invention, and changes in the arrangement and type of components may be made without departing from the spirit or scope of the present invention. As illustrated, the system 100 of FIG. 1 includes a local area network (LAN) / wide area network (WAN)-(network) 110, a wireless network 108, client computers 102-105, a visualization server computer 116, a data source server computer 118, and the like.

[0031] At least one embodiment of the client computers 102-105 is described in more detail below in connection with FIG. 2. In one embodiment, at least some of the client computers 102-105 may operate on one or more wired or wireless networks, such as the networks 108 or 110. In general, the client computers 102-105 may include virtually any computer capable of communicating over a network to send and receive information, perform various online activities, and perform offline actions. In one embodiment, one or more of the client computers 102-105 may be configured to operate within and perform various services for a business or other entity. For example, the client computers 102-105 may be configured to operate as a web server, a firewall, a client application, a media player, a mobile phone, a game console, a desktop computer, and the like. However, the client computers 102-105 are not limited to these services and may also be used, for example, for end-user computing in other embodiments. It should be appreciated that more or fewer client computers (as shown in FIG. 1) may be included in a system as described herein, and thus the embodiments are not constrained by the number or type of client computers used.

[0032] Computers operating as client computers 102 may include computers that typically connect using wired or wireless communication media, such as personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, network PCs, and the like. In some embodiments, client computers 102-105 may include virtually any portable computer capable of connecting to and receiving information from another computer, such as laptop computer 103, mobile computer 104, tablet computer 105, and the like. However, portable computers may also include other portable computers, such as, but not limited to, mobile phones, display pagers, radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), handheld computers, wearable computers, integrated devices that combine one or more of the preceding computers, and the like. Thus, client computers 102-105 typically range widely in terms of capabilities and features. Additionally, client computers 102-105 may access a variety of computing applications, including browsers or other web-based applications.

[0033] A web-enabled client computer may include a browser application configured to send requests and receive responses over the web. The browser application may be configured to receive and display graphics, text, multimedia, and the like, using virtually any web-based language. In one embodiment, the browser application is capable of displaying and sending messages using JavaScript, HyperText Markup Language (HTML), eXtensible Markup Language (XML), JavaScript Object Notation (JSON), Cascading Style Sheets (CSS), and the like, or combinations thereof. In one embodiment, a user of a client computer may use the browser application to perform various activities over a network (online). However, other applications may also be used to perform various online activities.

[0034] The client computers 102-105 may also include at least one other client application configured to receive or send content between other computers. The client application may include the ability to send or receive content, etc. The client application may further provide information that identifies itself, including type, capabilities, name, etc. In one embodiment, the client computers 102-105 may uniquely identify themselves via any of a variety of mechanisms, including an Internet Protocol (IP) address, a phone number, a Mobile Identification Number (MIN), an Electronic Serial Number (ESN), a client certificate, or other device identifier. Such information may be provided, such as in one or more network packets transmitted between the other client computers, the visualization server computer 116, the data source server computer 118, or other computers.

[0035] The client computers 102-105 may further be configured to include a client application that allows an end user to log into an end user account that may be managed by another computer, such as the visualization server computer 116, the data source server computer 118, etc. In one non-limiting example, such an end user account may be configured to allow the end user to manage one or more online activities including, in one non-limiting example, project management, software development, systems administration, configuration management, search activity, social networking activity, browsing various websites, communicating with other users, etc. The client computers may also be arranged to allow the user to view reports, interactive user interfaces, or results provided by the visualization server computer 116, the data source server computer 118, etc.

[0036] Wireless network 108 is configured to couple client computers 103-105 and their components with network 110. Wireless network 108 may include any of a variety of wireless sub-networks that may further overlay standalone ad-hoc networks, etc., to provide infrastructure-oriented connectivity for client computers 103-105. Such sub-networks may include mesh networks, wireless local area network (WLAN) networks, cellular networks, etc. In one embodiment, the system may include multiple wireless networks.

[0037] Wireless network 108 may further include autonomous systems of terminals, gateways, routers, etc., connected by wireless radio links, etc. These connectors are configured to move freely and randomly and organize themselves arbitrarily, allowing the topology of wireless network 108 to change rapidly.

[0038] The wireless network 108 may further use multiple access technologies including second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G) wireless access for cellular systems, WLAN, wireless router (WR) mesh, etc. Access technologies such as 2G, 3G, 4G, 5G, and future access networks may enable wide area coverage for mobile computers such as the client computers 103-105 with various mobility. In one non-limiting example, the wireless network 108 may enable wireless connectivity via wireless network access such as Global System for Mobil communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), etc. In essence, wireless network 108 may include virtually any wireless communication mechanism by which information can travel between client computers 103-105 and another computer, network, cloud-based network, cloud instance, etc.

[0039] Network 110 is configured to couple the network computers to other computers, including visualization server computer 116, data source server computer 118, client computer 102, and client computers 103-105, via wireless network 108, or the like. Network 110 may use any form of computer-readable media to communicate information from one electronic device to another. Network 110 may also include the Internet, as well as direct connections via local area networks (LANs), wide area networks (WANs), universal serial bus (USB) ports, Ethernet ports, other forms of computer-readable media, or any combination thereof. In a set of interconnected LANs, including those based on different architectures and protocols, routers act as links between the LANs, allowing them to send messages to one another. Additionally, while communication links within a LAN typically include twisted wire pairs or coaxial cables, communication links between networks may utilize other carrier mechanisms including analog telephone lines, fully or partially dedicated digital lines including T1, T2, T3, and T4, or wireless links including, for example, E-carrier, Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), satellite links, or other communication links known to those skilled in the art. Furthermore, the communication links may further use any of a variety of digital signaling techniques including, for example, but not limited to, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, etc. Additionally, remote computers and other associated electronic devices may be remotely connected to either the LAN or WAN via modems and temporary telephone links. In one embodiment, the network 110 may be configured to transfer information in the Internet Protocol (IP).

[0040] Additionally, communication media typically embodies computer readable instructions, data structures, program modules, or other transport mechanism and includes any information non-transitory or transitory delivery media. By way of example, communication media includes wired media such as twisted pair, coaxial cable, fiber optics, wave guides, and other wired media, and wireless media such as acoustic, RF, infrared and other wireless media.

[0041] Additionally, one embodiment of the visualization server computer 116, the data source server computer 118, etc., is described in more detail below in conjunction with FIG. 3. Although FIG. 1 illustrates the visualization server computer 116, the data source server computer 118, etc., as each being a single computer, the innovations or embodiments are not so limited. For example, one or more functions of the visualization server computer 116, the data source server computer 118, etc., may be distributed across one or more separate network computers. Additionally, in one or more embodiments, the visualization server computer 116, the data source server computer 118, etc. may be implemented using multiple network computers. Additionally, in one or more of the various embodiments, the visualization server computer 116, the data source server computer 118, etc., may be implemented using one or more cloud instances in one or more cloud networks. Thus, these innovations and embodiments should not be construed as limited to a single environment, and other configurations and other architectures are also contemplated.

[0042] Exemplary Client Computer

[0043] 2 illustrates an embodiment of a client computer 200 that may include many more or fewer components than those illustrated. Client computer 200 may represent, for example, one or more embodiments of the mobile computer or client computer illustrated in FIG.

[0044] The client computer 200 may include a processor 202 in communication with a memory 204 via a bus 228. The client computer 200 may also include a power supply 230, a network interface 232, an audio interface 256, a display 250, a keypad 252, an illuminator 254, a video interface 242, an input / output interface 238, a haptic interface 264, a global positioning system (GPS) receiver 258, an open air gesture interface 260, a temperature interface 262, a camera(s) 240, a projector 246, a pointing device interface 266, a processor-readable fixed storage device 234, and a processor-readable removable storage device 236. The client computer 200 may optionally communicate with a base station (not shown) or directly with another computer. Also, in one embodiment, although not shown, a gyroscope may be used within the client computer 200 to measure or maintain the orientation of the client computer 200.

[0045] The power supply 230 may provide power to the client computer 200. A rechargeable or non-rechargeable battery may be used to provide the power. Power may also be provided by an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the battery.

[0046] The network interface 232 includes circuitry for coupling the client computer 200 to one or more networks and is constructed for use with one or more communications protocols and technologies, including, but not limited to, protocols and technologies implementing any portion of the OSI model for mobile communications (GSM), CDMA, time division multiple access (TDMA), UDP, TCP / IP, SMS, MMS, GPRS, WAP, UWB, WiMax, SIP / RTP, GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or any of a variety of other wireless communication protocols. The network interface 232 may be known as a transceiver, transceiver device, or network interface card (NIC).

[0047] Audio interface 256 may be arranged to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 256 may be coupled to a speaker and microphone (not shown) to enable communication with others or to generate audio acknowledgments for some actions. The microphone in audio interface 256 may also be used for input to or control of client computer 200, for example using voice recognition, sound-based touch detection, etc.

[0048] Display 250 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LCD), organic LED (OLED), or any other type of reflective or light transmissive display that may be used with a computer. Display 250 may also include a touch interface 244 arranged to receive input from an object such as a stylus or the finger of a human hand, and may use resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other technologies for sensing touch or gestures.

[0049] Projector 246 may be a remote handheld projector or an integrated projector capable of projecting an image onto a remote wall or other reflective object, such as a remote screen.

[0050] Video interface 242 may be arranged to capture video images, such as still photographs, video segments, infrared video, etc. For example, video interface 242 may be coupled to a digital video camera, a webcam, etc. Video interface 242 may include a lens, an image sensor, and other electronics. The image sensor may include a complementary metal-oxide semiconductor integrated circuit (CMOS), a charge-coupled device (CCD), or any other integrated circuit for sensing light.

[0051] Keypad 252 may include any input device arranged to receive input from a user. For example, keypad 252 may include a push button numeric dial, or a keyboard. Keypad 252 may also include command buttons associated with selecting and sending images.

[0052] The illuminator 254 may provide a status indication or provide light. The illuminator 254 may remain active for a particular period of time or in response to an event message. For example, when activated, the illuminator 254 may backlight the buttons of the keypad 252 and remain on while the client computer is powered on. The illuminator 254 may also backlight these buttons in various patterns when certain actions are performed, such as dialing another client computer. The illuminator 254 may also cause light sources positioned within the transparent or translucent case of the client computer to illuminate in response to actions.

[0053] Additionally, client computer 200 may also include a hardware security module (HSM) 268 to provide additional tamper-resistant safeguards for generating, storing, or using security / cryptographic information, such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some embodiments, the hardware security module may be used to support one or more standard public key infrastructures (PKIs), the hardware security module may be used to generate, manage, or store key pairs, etc. In some embodiments, HSM 268 may be a standalone computer, while in other cases, HSM 268 may be arranged as a hardware card that can be added to the client computer.

[0054] The client computer 200 may also include an input / output interface 238 for communicating with external peripheral devices or other computers, such as other client computers and network computers. Peripheral devices may include audio headsets, virtual reality headsets, display screen glasses, remote speaker systems, remote speaker and microphone systems, etc. The input / output interface 238 may utilize one or more technologies, such as Universal Serial Bus (USB), infrared, WiFi, WiMax, Bluetooth, etc.

[0055] The input / output interface 238 may also include sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation scales, etc.), etc. The sensors may be one or more hardware sensors that collect or measure data external to the client computer 200.

[0056] The haptic interface 264 may be arranged to provide haptic feedback to the user of the client computer. For example, the haptic interface 264 may be used to cause the client computer 200 to vibrate in a particular manner when another user of the computer is calling. The temperature interface 262 may be used to provide a temperature measurement input or a temperature change output to the user of the client computer 200. The open air gesture interface 260 may sense the physical gestures of the user of the client computer 200, for example, by using a single or stereo video camera, radar, a gyro sensor inside a computer held or worn by the user, etc. The camera 240 may be used to track the physical eye movements of the user of the client computer 200.

[0057] The GPS transceiver 258 can determine the physical coordinates of the client computer 200 on the surface of the Earth, which typically output a location as latitude and longitude values. The GPS transceiver 258 can also further determine the physical location of the client computer 200 on the surface of the Earth using other geopositioning mechanisms, including, but not limited to, triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Enhanced Timing Advance (ETA), Base Station Subsystem (BSS), and the like. It is understood that under different conditions, the GPS transceiver 258 can determine the physical location of the client computer 200. However, in one or more embodiments, the client computer 200, through other components, may provide other information, including, for example, a Media Access Control (MAC) address, an IP address, and the like, that can be used to determine the physical location of the client computer.

[0058] In at least one of various embodiments, applications such as the operating system 206, other client applications 224, and web browser 226 may be configured to utilize the geolocation information to select one or more local characteristics, such as a time zone, language, currency, calendar format, etc. The local characteristics may be used in display objects, data models, data objects, user interfaces, reports, and internal processes or databases. In at least one of various embodiments, the geolocation information used to select localized information may be provided by GPS 258. Also, in some embodiments, the geolocation information may include information provided using one or more geolocation protocols over a network, such as wireless network 108 or network 111.

[0059] The human interface components may be peripheral devices physically separate from the client computer 200, allowing remote input to or output from the client computer 200. For example, information routed as described herein through a human interface component such as the display 250 or keyboard 252 may instead be routed through the network interface 232 to an appropriate remotely located human interface component. Examples of human interface peripheral components that may be remote may include, but are not limited to, audio devices, pointing devices, keypads, displays, cameras, projectors, and the like. These peripheral components may communicate over pico networks such as Bluetooth, Zigbee™, and the like. One non-limiting example of a client computer having such peripheral human interface components is a wearable computer, which may include a remote pico projector along with one or more cameras that communicate remotely with the separately located client computer to sense a user's gestures relative to a portion of an image projected by the pico projector on a reflective surface such as a wall or the user's hand.

[0060] The client computer may include a web browser application 226 configured to receive and send web pages, web-based messages, graphics, text, multimedia, and the like. The browser application of the client computer may use virtually any programming language, including wireless application protocol messages (WAP), and the like. In one or more embodiments, the browser application may use Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WML script, JavaScript, Standard Generalized Markup Language (SGML), HyperText Markup Language (HTML), eXtensible Markup Language (XML), HTML5, and the like.

[0061] The memory 204 may include RAM, ROM, or other types of memory. The memory 204 illustrates an example of a computer-readable storage medium for storing information such as computer-readable instructions, data structures, program modules, or other data. The memory 204 may store a BIOS 208 for controlling low-level operation of the client computer 200. The memory may also store an operating system 206 for controlling the operation of the client computer 200. It will be appreciated that this component may include general-purpose operating systems such as UNIX, LINUX, or specialized client computer communications operating systems such as Windows Phone, Symbian, or other operating systems. The operating system may include or interface with a Java Virtual Machine module that allows control of hardware components or operation of the operating system via Java application programs.

[0062] The memory 204 may further include one or more data stores 210 that may be utilized by the client computer 200 to store, among other things, applications 220 or other data. For example, the data store 210 may also be used to store information describing various capabilities of the client computer 200. The information may then be provided to another device or computer based on any of a variety of methods, including as part of a header during a communication, on request, etc. The data store 210 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, etc. The data store 210 may further include program code, data, algorithms, etc. for use by a processor, such as the processor 202, to execute and perform actions. In one embodiment, at least a portion of the data store 210 may also be stored in another component of the client computer 200, including, but not limited to, a non-transitory processor-readable removable storage device 236, a processor-readable non-removable storage device 234, or even external to the client computer.

[0063] Applications 220 may include computer-executable instructions that, when executed by client computer 200, transmit, receive, or otherwise process instructions and data. Applications 220 may include, for example, a client visualization engine 222, other client applications 224, a web browser 226, etc. The client computer may be arranged to communicate with one or more servers.

[0064] Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice Over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, visualization applications, and the like.

[0065] Additionally, in one or more embodiments (not shown), the client computer 200 may include embedded logic hardware, such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Array Logic (PAL), or the like, or a combination thereof, instead of a CPU. The embedded logic hardware device may directly execute its embedded logic to perform actions. Also, in one or more embodiments (not shown), the client computer 200 may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, the one or more microcontrollers may directly execute their embedded logic to perform actions and access their internal memory and their external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as a system-on-a-chip (SOC).

[0066] Exemplary Network Computer

[0067] 3 illustrates one embodiment of a network computer 300 that may be included in a system implementing one or more of the various embodiments. The network computer 300 may include many more or fewer components than those illustrated in FIG. 3. However, the illustrated components are sufficient to disclose an example embodiment for implementing these innovations. The network computer 300 may represent, for example, at least one example of the visualization server computer 116, the data source server computer 118, etc. of FIG. 1.

[0068] A network computer such as network computer 300 may include a processor 302 that may communicate with memory 304 via a bus 328. In some embodiments, processor 302 may be comprised of one or more hardware processors, or one or more processor cores. In some cases, one or more of the one or more processors may be specialized processors designed to perform one or more specialized actions as described herein. Network computer 300 also includes a power supply 330, a network interface 332, an audio interface 356, a display 350, a keyboard 352, an input / output interface 338, a processor-readable fixed storage device 334, and a processor-readable removable storage device 336. Power supply 330 may provide power to client computer 300.

[0069] The network interface 332 includes circuitry for coupling the network computer 300 to one or more networks and is constructed for use with one or more communications protocols and technologies, including, but not limited to, protocols and technologies implementing any portion of the Open Systems Interconnection model (OSI model), global system for mobile communication (GSM), code division multiple access (CDMA), time division multiple access (TDMA), user datagram protocol (UDP), transmission control protocol / Internet protocol (TCP / IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), general packet radio service (GPRS), WAP, ultra-wide band (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), Session Initiation Protocol / Real-time Transport Protocol (SIP / RTP), or any of a variety of other wired and wireless communication protocols. The network interface 332 is sometimes known as a transceiver, transceiver device, or network interface card (NIC). Network computer 300 may optionally communicate directly with a base station (not shown), or with another computer.

[0070] Audio interface 356 is arranged to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 356 may be coupled to a speaker and microphone (not shown) to enable communication with others or to generate audio acknowledgments for some actions. The microphone in audio interface 356 may also be used for input to or control of network computer 300, for example using voice recognition.

[0071] Display 350 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LCD), organic LED (OLED), or any other type of reflective or light transmissive display that may be used with a computer. In some embodiments, display 350 may be a handheld projector or picoprojector that can project an image onto a wall or other object.

[0072] Network computer 300 may also include an input / output interface 338 for communicating with external devices or computers not shown in Figure 3. Input / output interface 338 may utilize one or more wired or wireless communication technologies, such as USB, Firewire, WiFi, WiMax, Thunderbolt, infrared, Bluetooth, Zigbee, serial ports, parallel ports, etc.

[0073] The input / output interface 338 may also include sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation scales, etc.), and the like. The sensors may be one or more hardware sensors that collect or measure data external to the network computer 300. The human interface components may be physically separate from the network computer 300, allowing for remote input to or output from the network computer 300. For example, information routed as described herein through a human interface component such as the display 350 or keyboard 352 may instead be routed through the network interface 332 to an appropriate human interface component located elsewhere on the network. The human interface components include any components that allow the computer to receive input from or send output to a human user of the computer. Thus, a pointing device such as a mouse, stylus, trackball, etc. may communicate through the pointing device interface 358 to receive user input.

[0074] The GPS transceiver 340 can determine the physical coordinates of the network computer 300 on the surface of the Earth, which typically output a location as latitude and longitude values. The GPS transceiver 340 can also use other geopositioning mechanisms, including but not limited to triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Enhanced Timing Advance (ETA), Base Station Subsystem (BSS), and the like, to further determine the physical location of the network computer 300 on the surface of the Earth. It is understood that under different conditions, the GPS transceiver 340 can determine the physical location of the network computer 300. However, in one or more embodiments, the network computer 300, through other components, may provide other information, including, for example, a Media Access Control (MAC) address, an IP address, and the like, that can be used to determine the physical location of the network computer.

[0075] In at least one of various embodiments, applications such as operating system 306, evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329, etc. may be configured to use the geolocation information to select one or more local characteristics, such as a time zone, a language, a currency, a currency format, a calendar format, etc. The local characteristics may be used in user interfaces, dashboards, visualizations, reports, and internal processes or databases. In at least one of various embodiments, the geolocation information used to select the localization information may be provided by GPS 340. Also, in some embodiments, the geolocation information may include information provided using one or more geolocation protocols over a network, such as wireless network 108 or network 111.

[0076] The memory 304 may include random access memory (RAM), read only memory (ROM), or other types of memory. The memory 304 illustrates an example of a computer-readable storage medium for storing information such as computer-readable instructions, data structures, program modules, or other data. The memory 304 stores a basic input / output system (BIOS) 308 for controlling low-level operation of the network computer 300. The memory may also store an operating system 306 for controlling the operation of the network computer 300. It will be appreciated that this component may include a general-purpose operating system, such as a version of UNIX or LINUX™, or a specialized operating system, such as Microsoft's Windows operating system or Apple's OSX operating system. The operating system may include or interface with one or more virtual machine modules, such as a Java virtual machine module that allows control of hardware components or operation of the operating system via Java application programs. Other runtime environments may be included as well.

[0077] Memory 304 may further include one or more data storage devices 310 that may be utilized by network computer 300 to store, among other things, applications 320 or other data. For example, data storage device 310 may also be used to store information describing various capabilities of network computer 300. The information may then be provided to another device or computer based on any of a variety of methods, including being sent as part of a header during a communication, being sent upon request, etc. Data storage device 310 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, etc. Data storage device 310 may further include program code, data, algorithms, etc. for use by a processor, such as processor 302, to execute and perform actions, such as those described below. In one embodiment, at least a portion of data storage device 310 may also be stored in another component of network computer 300, including, but not limited to, non-transitory media in processor readable removable storage device 336, processor readable non-transitory storage device 334, or any other computer readable storage device within network computer 300, or even external to network computer 300. Data storage device 310 may include, for example, data models 314, data sources 316, visualization models 318, evaluation models 319, etc.

[0078] The applications 320 may include computer-executable instructions that, when executed by the network computer 300, send, receive, or otherwise process messages (e.g., SMS, Multimedia Messaging Service (MMS), Instant Message (IM), email, or other messages), audio, video, and other enabled electronic communications with another user of another mobile computer. Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice Over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and the like. The applications 320 may include an evaluation engine 322, a visualization engine 324, a modeling engine 326, other applications 329, and the like, which may be arranged to perform actions for the embodiments described below. In one or more of the various embodiments, one or more of the applications may be implemented as a module or component of another application. Additionally, in one or more of the various embodiments, an application may be implemented as an operating system extension, module, plug-in, and the like.

[0079] Further, in one or more of various embodiments, the evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329, etc. may be operable in a cloud-based computing environment. In one or more of various embodiments, these applications, and others including the management platform, may run in virtual machines or virtual servers that may be managed in the cloud-based computing environment. In one or more of various embodiments, in this context, applications may flow from one physical network computer to another in the cloud-based environment depending on performance and scaling considerations that are automatically managed by the cloud computing environment. Similarly, in one or more of various embodiments, virtual machines or virtual servers dedicated to the evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329 may be automatically provisioned and decommissioned.

[0080] Also, in one or more of various embodiments, the evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329, etc. may be located on virtual servers operating in a cloud-based computing environment, rather than being tied to one or more particular physical network computers.

[0081] Additionally, network computer 300 may also include a hardware security module (HSM) 360 to provide additional tamper-resistant safeguards for generating, storing, or using security / cryptographic information, such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some embodiments, the hardware security module may be used to support one or more standard public key infrastructures (PKIs), the hardware security module may be used to generate, manage, or store key pairs, etc. In some embodiments, HSM 360 may be a standalone network computer, while in other cases HSM 360 may be arranged as a hardware card that can be added to a client computer.

[0082] Additionally, in one or more embodiments (not shown), the client computer 300 may include embedded logic hardware such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Array Logic (PAL), or the like, or a combination thereof, instead of a CPU. The embedded logic hardware device may directly execute its embedded logic to perform actions. Also, in one or more embodiments (not shown), the client computer 200 may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, the one or more microcontrollers may directly execute their embedded logic to perform actions and access their internal memory and their external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as a system-on-a-chip (SOC).

[0083] Exemplary Logical System Architecture

[0084] 4 illustrates a logical architecture of a system 400 for surfacing visualization mirages, according to one or more of various embodiments. In one or more of various embodiments, system 400 may be comprised of various components, including one or more modeling engines, such as modeling engine 402, one or more visualization engines, such as visualization engine 404, one or more visualizations, such as visualization 406, one or more data sources, such as data source 410, one or more visualization model stores, such as visualization model store 408, or one or more evaluation engines, such as evaluation engine 412.

[0085] In one or more of various embodiments, the modeling engine 402 may be arranged to allow a user to design one or more visualization models, which may be provided to the visualization engine 404. Thus, in one or more of various embodiments, the visualization engine 404 may be arranged to generate one or more visualizations based on the visualization models.

[0086] In one or more of the various embodiments, the modeling engine may be arranged to access one or more data sources, such as data source 410. In some embodiments, the modeling engine may be arranged to include a user interface that allows a user to view various data source information, data objects, etc., to design visualization models that can be used to generate visualizations of information stored in the data sources.

[0087] Thus, in some embodiments, the visualization model may be designed to provide visualization including charts, plots, graphs, tables, graphics, styling, explanatory text, interactive elements, user interface features, etc. In some embodiments, a graphical user interface may be provided to a user, allowing the user to interactively design the visualization model such that various elements or display objects in the visualization model are associated with data from one or more data sources, such as data source 410.

[0088] In one or more of various embodiments, a data source, such as data source 410, may include one or more of a database, a data store, a file system, etc., which may be located locally or remotely. In some embodiments, the data source may be provided by another service over a network. In some embodiments, there may be one or more components (not shown) that filter the data in the data source or otherwise provide an administrative view or access to the data.

[0089] In one or more of various embodiments, the visualization models may be stored in one or more data stores, such as visualization model store 408. In this example, for some implementations, visualization model store 408 represents one or more databases, file systems, or the like for storing, persisting, or indexing the visualization models.

[0090] In one or more of the various embodiments, a visualization engine, such as visualization engine 404, may be arranged to analyze or otherwise interpret data from the visualization model and data sources to generate one or more visualizations that may be displayed to a user.

[0091] In one or more of various embodiments, a rating engine, such as rating engine 412, may be arranged to rate or otherwise assess the visualization. Thus, in some embodiments, the rating engine may be arranged to automatically perform one or more actions to identify possible mirages or artifacts in the visualization that result in a misleading visualization. In some embodiments, the rating engine may be arranged to automatically ingest additional data from data sources related to the visualization and assess its relevance using a statistical model, such as a rating model.

[0092] 5A and 5B show an example of how visualization can be misleading.

[0093] FIG. 5A illustrates a representation of a portion of a visualization 500 for surfacing a visualization mirage, according to one or more of various embodiments. In this example, the visualization 500 may be considered a radar chart. In some cases, a radar chart may be used to compare the skills of job candidates. In this example, the visualization 500 illustrates the job skills of a hypothetical job candidate, where each axis (e.g., A, B, C, D, E, and F) represents a skill level of a particular job skill. Also in this example, where the line 502 intersects with the axis, it indicates the job candidate's skill level of a particular job skill. Thus, in this example, the job candidate has a high level of competency in job skills B, C, and D, and a lower level of competency in job skills A, F, and E.

[0094] Thus, at first glance, visualization 500 may appear to indicate that a job candidate may have a focused skill set, as the job candidate appears to have skills that appear to be concentrated in the upper right portion of the visualization.

[0095] 5B illustrates a representation of a portion of visualization 504 for surfacing a visualization mirage, according to one or more of various embodiments. In this example, similar to visualization 500, visualization 504 may be thought of as a radar chart. Similarly, in this example, visualization 504 illustrates the job skills of a hypothetical job candidate, where each axis (e.g., A, B, C, D, E, and F) represents a skill level for a particular job skill. Also in this example, where line 506 intersects an axis, it indicates the job candidate's skill level for a particular job skill. Thus, similar to visualization 500, visualization 504 illustrates that the job candidate has a high level of competency in job skills B, C, and D, and a lower level of competency in job skills A, F, and E.

[0096] However, in this example, axis C and axis F have been swapped in position in the visualization. Thus, in contrast to visualization 500, visualization 504 appears to indicate that job candidates may have a more diverse (e.g., less concentrated) skill set because the job candidates appear to have skills that appear to be more evenly distributed in the visualization.

[0097] In this example, the appearance of visualization 500 and visualization 504 is very different, even though the underlying data may be the same. Thus, in this example, there may be one or more visualization mirages.

[0098] Similar to Figures 5A and 5B, Figures 6A and 6B show another example of how mirages can be included in an otherwise reasonable looking visualization.

[0099] 6A shows a portion of a visualization 600 that includes a bar graph comparing home prices in 2001 (represented by axis 602) to home prices in 2008 versus year categories represented by axis 604. Thus, in this example, the appearance of visualization 600 appears to show that prices increased 300% from 2001 to 2008 due to the relative sizes of the bars used in the graph.

[0100] 6B shows a portion of a visualization 610 that includes a bar graph comparing home prices in 2001 (represented by axis 612) to home prices in 2008 versus year categories represented by axis 614. Thus, in this example, the appearance of visualization 614 appears to show that prices barely increased from 2001 to 2008 due to the relative sizes of the bars used in the graphs.

[0101] In this example, the visual appearance of Figures 6A and 6B seem to suggest quite different conclusions. Figure 6A seems to indicate that the price changed dramatically, while Figure 6B seems to indicate that the price did not increase dramatically. The point is that in this example, as well as the examples shown in Figures 5A and 5B, the visualizations can have very different appearances, even though the underlying data or type of visualization may be similar. A visualization that counters this type of effect (e.g., mirage) may be considered a robust visualization.

[0102] It is noted that one skilled in the art would understand that a visualization model or visualization engine may be arranged to generate many different types of visualizations for various purposes depending on the design goals of an author, user, or organization. Visualization 500, visualization 504, visualization 600, and visualization 610 are presented here as non-limiting examples to help clarify the description of these innovations. One skilled in the art would understand that these examples are at least sufficient to disclose the innovations herein, and that a visualization engine or visualization model may be arranged to generate many different visualizations in many domains and for many different purposes.

[0103] 7 illustrates a logical representation of a portion of a rating system 700 according to one or more of various embodiments. In one or more of various embodiments, the system 700 may include one or more components, such as a rating engine 702, a rating model 704, a visualization model 706, a data source 708, a visualization 710, and rating results 712. In some embodiments, the rating results, such as the rating results 712, may be arranged to include additional information, such as a rating score 714, report information 716, etc.

[0104] In one or more of various embodiments, the rating engine 702 may be arranged to rate the visualization 710 based on the rating model 704. In one or more of various embodiments, the rating model may be arranged to include one or more heuristic or machine learning assessors that may be executed to rate the visualization.

[0105] As discussed herein, the rating engine may be arranged to use one or more rating models to provide information for one or more reports regarding how well a given rating model matches (or classifies) a visualization. Thus, in this example, the rating results 712 include scores, such as rating score 714 and report information 716.

[0106] In one or more of various embodiments, the evaluation model may be arranged to provide an evaluation score, such as a probability score, a confidence score, etc., that represents the quality of the evaluation. In some embodiments, the evaluation engine may be arranged to execute or apply the evaluation model to perform various actions to assess the visualization, the visualization model, or the data source to determine whether a given visualization may include one or more visualization mirages.

[0107] In one or more of the various embodiments, different rating models may use different scoring criteria. Thus, in some embodiments, the rating engine may be arranged to weight or normalize the rating scores provided by the different rating models. In some embodiments, specific normalization or weighting rules for normalizing or weighting the rating scores of the rating models may be provided via configuration information to take into account local conditions.

[0108] In one or more of the various embodiments, an evaluation model may be designed or tuned to assess one or more statistical characteristics of data associated with a potential mirage. Thus, in one or more of the various embodiments, the evaluation engine may be arranged to apply one or more evaluation models to assess whether data associated with the visualization under evaluation has one or more of the statistical characteristics targeted by the evaluation model. In some embodiments, the evaluation engine may be arranged to provide an evaluation score as a form of a self-grade that represents the extent to which the data associated with the visualization matches the statistical characteristics that the evaluation model may have been designed to match or otherwise assess.

[0109] Thus, in one or more of various embodiments, the rating engine may be arranged to use user activity information or user feedback to automatically build a rating model profile that can be used to modify or customize the rating report. For example, if users of an organization consistently report discrepancies between visualizations and rating results, the rating engine may be arranged to introduce weighting rules that increase or decrease the effective rating score used to rank the organization's rating results based on user feedback information.

[0110] In one or more of various embodiments, when a user selects a visualization for a mirage evaluation, the evaluation engine determines one or more evaluation models and applies them to the visualization being evaluated along with any associated visualization models or data sources. Accordingly, non-limiting examples of evaluation models are discussed below. For brevity and clarity, this discussion is limited to a few examples, but one of ordinary skill in the art will understand that other evaluation models are contemplated that may incorporate other or additional evaluation strategies.

[0111] In one or more of various embodiments, under an analytical framework known as Algebraic Visualization Design (AVD), trivial changes to the data underlying a visualization (such as shuffling the row order of the input data) should result in subtle changes in the resulting visualization, and significant changes in the appearance of the visualization should only occur as a result of corresponding significant data changes in the backing data. Those skilled in the art will appreciate that AVD formulates these assertions via commutativity relations that describe the properties of visualizations that are effective across possible data transformations (not shown).

[0112] In one or more of various embodiments, failures of AVD assertions may result in "hallucinations" (visualizations that appear dramatically different despite being supported by similar or identical data, such as in Figures 5A, 5B, 6a, or 6B) and "confusion" (visualizations that appear identical despite being supported by dramatically different data). In some cases, visualizations may be completely non-reactive to their supporting data and act as mere figure decoration, creating what may be called visualization "irrational." In some cases, these AVD failures may be directly coupled to visualization mirages (to result in visualizations that may be fragile, non-robust, or non-reactive). However, AVDs provide an analytical language of data manipulation and visualization specifications that allows for automated assessment by evaluation models. Thus, in some embodiments, AVDs may provide a useful framework for designing tests that may detect failures that require little domain knowledge. For example, in some cases, an evaluation model based on AVDs may simply trigger trivial or significant changes in the data and check for corresponding changes in the resulting visualizations.

[0113] In one or more of various embodiments, the evaluation engine may be arranged to use metamorphic testing as a mechanism to validate or otherwise assess individual visualizations. In some embodiments, the evaluation engine may be arranged to use evaluation models that can perform metamorphic operations (changing data and changing design specifications). Thus, in some embodiments, the evaluation engine may be able to assess a wide range of types of visualizations without knowing too much about the visualization being rendered.

[0114] In one or more of various embodiments, a change in the order of the input data should not change the rendered visualization. Thus, in some embodiments, the evaluation model may be arranged to use a detection technique, such as a pixel differencing algorithm, that defines the maximum number of pixels that differ between the rendered images based on the order of the input rows. Thus, in some embodiments, an evaluation model that provides this type of test may enable the detection of overplotting. Note that in some embodiments, not all overplotting necessarily indicates visualization mirage, but alerting the user to its presence may be useful across many visualization types.

[0115] In some embodiments, the apparent patterns in the visualization should be robust; that is, certain relationships should continue to exist across minor changes. Thus, one or more evaluation models may be arranged to focus on providing a mirage detector for bar charts. In some embodiments, such evaluation models may test for variability in a relatively parameter-free manner across a wide range of data distributions and complexities. In some embodiments, the evaluation models may be arranged to identify which input rows to modify by creating a concept of posterior provenance that links each aggregate mark to the input tuples that describe it. Thus, the evaluation models may be able to identify when the visualization is likely to rely on outliers, or when a small number of divergent records are driving differences between aggregates.

[0116] In one or more of various embodiments, the evaluation model may be arranged to assess aggregates that may be commonly and advantageously useful for summarizing information. However, in some cases, aggregates may mask problems in the data, such as the number of distinct records, sampling issues, repeated records, etc. Thus, in some embodiments, the evaluation model may be arranged to assess the robustness of measures in the context of potentially tainted data. In some embodiments, the evaluation model may be configured to define a minimum number of records that constitute an aggregate, and may reduce the number of records that constitute all other marks to that minimum number through sampling without replacement. In some embodiments, the evaluation model may use a randomization procedure to explore aggregates in the visualization. Thus, if all aggregates have similar sample sizes, and this sample size is large enough, and the aggregation method is sufficiently robust against extreme values, this procedure should result in reasonably similar visualizations. Thus, in some embodiments, the evaluation model may be arranged to detect variability caused by sampling issues and other issues related to the number of distinct records.

[0117] 8 illustrates a logical representation of a portion of a user interface 800 for surfacing a visualization mirage, according to one or more of various embodiments. In some embodiments, user interface 800 may be arranged to include one or more panels, such as panel 802, panel 804, panel 806, panel 808, description 810, etc.

[0118] In this example, for one or more of the various embodiments, panel 802 may be arranged to display a ranked list of mirage rating information for the visualizations as rated by the rating engine. In some embodiments, each report item may include information describing the type of mirage that may be suspected, such as a rating score, if available.

[0119] Similarly, in some embodiments, if the rating engine is unable to determine a potential mirage, panel 802 may be arranged to display a statement to that effect.

[0120] In one or more of various embodiments, panel 804 may be arranged to include additional explanations or visualizations that may relate to the visualizations rated by the rating engine. In some embodiments, a panel such as panel 806 may include additional explanatory information related to the detected mirage, or one or more visualizations that may help explain the mirage to a user. Also, in some embodiments, a panel such as panel 808 may include the visualizations being rated, or other visualizations that may help a user "see" the effect of the determined mirage. For example, panel 808 may include one or more alternative visualizations that help show the effect of the detected mirage.

[0121] In one or more of various embodiments, the assessment engine, visualization engine, etc. may be arranged to use specific visualization models or explanatory text / narrative templates for different assessment models. In some embodiments, the visualization models or explanatory / narrative templates for assessment models may be defined in configuration information. Thus, the reporting information (e.g., explanatory / narrative text) or visualizations included in user interface 800 may be tailored to a particular organization, locale, etc.

[0122] In one or more of various embodiments, the evaluation engine or visualization engine may be arranged to determine some or all of the content or styling for the user interface 800 via the configuration information. Thus, in some embodiments, panel layouts, explanatory text, templates, visualizations, etc. may be configured differently depending on the local situation.

[0123] Generalized behavior

[0124] 9-12 illustrate generalized operations for surfacing a visualization mirage, according to one or more of the various embodiments. In one or more of the various embodiments, the processes 900, 1000, 1100, and 1200 described in conjunction with FIGS. 9-12 may be implemented by or executed on one or more processors on a single network computer (or network monitoring computer), such as network computer 300 of FIG. 3. In other embodiments, these processes, or portions thereof, may be implemented by or executed on multiple network computers, such as network computer 300 of FIG. 3. In still other embodiments, these processes, or portions thereof, may be implemented by or executed on one or more virtualized computers, such as in a cloud-based environment. However, the embodiments are not limited thereto, and various combinations of network computers, client computers, and the like may be utilized. Additionally, in one or more of the various embodiments, the processes described in conjunction with FIGS. 9-12 may be used to surface a visualization mirage, according to at least one of the various embodiments or architectures as described in conjunction with FIGS. 4-8. Further, in one or more of various embodiments, some or all of the actions performed by processes 900, 1000, 1100, and 1200 may be partially performed by the evaluation engine 322, the visualization engine 324, the modeling engine 326, etc.

[0125] 9 illustrates a general flowchart of a process 900 for surfacing a visualization mirage, according to one or more of various embodiments. After a start block, in one or more of various embodiments, at block 902, a visualization engine may be arranged to generate one or more visualizations based on one or more visualization models or data sources. As discussed above, a visualization system may be arranged to include one or more modeling engines, one or more data sources, one or more visualization engines, etc., and the one or more visualization engines may be arranged to generate visualizations based on one or more visualization models and data provided by one or more data sources.

[0126] At block 904, in one or more of the various embodiments, optionally, a rating engine may be arranged to determine one or more visualizations that may be of interest.

[0127] In one or more of various embodiments, a user or creator of a visualization may be able to select one or more visualizations for mirage analysis. For example, a user interface may be provided to allow the creator to reference one or more previously generated visualizations.

[0128] Similarly, in one or more of various embodiments, the rating engine may be arranged to automatically rate visualizations as they are created. In some embodiments, one or more visualizations may be determined for rating based on their membership in one or more classes or types of visualizations.

[0129] In one or more of various embodiments, the rating engine may be configured to apply various heuristics or filters to determine which visualizations to rate against the mirage. Thus, in some embodiments, the rating engine may be arranged to determine whether a visualization should be automatically evaluated against the mirage using one or more rules or instructions provided via the configuration information.

[0130] In some embodiments, the rating engine may be arranged to select or recommend one or more visualizations for rating based on criteria provided via the configuration information.

[0131] At block 906, in one or more of various embodiments, the rating engine may be arranged to analyze one or more visualizations of interest based on one or more rating models. In one or more of various embodiments, the rating engine may be arranged to execute the one or more rating models to evaluate the one or more visualizations to determine whether there may be hidden visualization mirages in the one or more visualizations.

[0132] At block 908, in one or more of various embodiments, the evaluation engine may be arranged to generate one or more mirage evaluation reports, which may include one or more visualizations attesting to the discovered mirages, if any. As mentioned above, in some embodiments, the evaluation report may include an interactive user interface that allows a user to review potential visualized mirages that may be discovered by the evaluation engine.

[0133] Control may then be returned to the calling process in one or more of the various embodiments.

[0134] FIG. 10 illustrates a flow chart of a process 1000 for evaluating visualizations to discover visualization mirages, according to one or more of various embodiments. After a start block, at block 1002, in one or more of various embodiments, an evaluation engine may be provided with one or more evaluation models and one or more visualizations. As described above, in some embodiments, one or more evaluation models may be defined to discover visualization mirages. Thus, in some embodiments, the evaluation engine may be configured to retrieve one or more evaluation models from a data store or the like. In some embodiments, the evaluation engine may be arranged to determine a particular evaluation model based on rules, conditions, or the like, which may be provided via configuration information.

[0135] At block 1004, in one or more of the various embodiments, the evaluation engine may be arranged to evaluate one or more evaluation models based on the marks, the visualization model, the data source, etc. In one or more of the various embodiments, the evaluation model may include or be associated with rules, conditions, computer readable instructions, etc. that may be executed or applied to evaluate if the visualization may include one or more visualization mirages.

[0136] At decision block 1006, in one or more of the various embodiments, if there are additional valuation models to evaluate, control may loop back to block 1004; otherwise, control may flow to block 908. In some embodiments, the valuation engine may be arranged to terminate the valuation early by omitting or skipping one or more valuation models. For example, in some embodiments, the valuation engine may be configured to cease evaluation of additional valuation models if a previously evaluated valuation model may be deemed to provide sufficient mirage analysis. In some embodiments, one or more valuation models may be redundant or debatable if one or more other valuation models identify or exclude some mirages.

[0137] In one or more of various embodiments, the rating engine may be arranged to associate a rating score with the rating results provided by the rating model. In some embodiments, the rating model may be arranged to provide a rating score indicating how well the rating model matches a particular visualization mirage.

[0138] Also, in one or more of the various embodiments, the rating engine may be arranged to maintain a separate set of rating scores that may be combined with the rating scores provided by each rating model used to identify visualized mirages. Thus, in some embodiments, an overall rating score associated with a discovered mirage may be a combination of the scores provided by the rating models and the scores provided by the rating engine.

[0139] For example, in some embodiments, the rating scores provided by the rating engine may be generated based on other factors or criteria not available or related to the rating model, in some embodiments, such factors or criteria may be based on user feedback, organizational requirements (e.g., configuration information), metrics related to organizational or community activities, etc.

[0140] At block 1008, in one or more of various embodiments, the rating engine may be arranged to filter or rank the results provided by the rating model. In one or more of various embodiments, the rating model may be arranged to provide one or more rating scores representing a self-assessment of the quality of the mirage determination.

[0141] Also, in one or more of the various embodiments, the rating engine may be arranged to modify the rating scores provided by the rating model to provide a final rating score.

[0142] Thus, in one or more of various embodiments, the rating engine may be arranged to remove one or more determined mirages based on filtering out one or more rating models based on rules, conditions, etc. For example, in some embodiments, the rating engine may be configured to automatically filter out one or more mirages associated with rating scores below a defined threshold.

[0143] Also, in one or more of various embodiments, other criteria or features of one or more of the visualization model, data source, data, user, organization, etc. may be incorporated into conditions, rules, etc. that may be executed to filter out one or more assessments or mirages. For example, in some embodiments, a filter may include conditions related to one or more user roles. Thus, for example, a filter may be configured to be more inclusive for some user roles or less inclusive for other user roles. Thus, in this example, the number of potential mirages included in the assessment report may be increased for data scientists or decreased for regular users.

[0144] In some embodiments, other criteria or features used for filtering may be based on a variety of sources including organizational preferences, user preferences, localization, user input, other configuration information, and the like.

[0145] Similarly, in one or more of various embodiments, the rating engine may be arranged to rank one or more discovered mirages based on various conditions, rules, criteria, or characteristics. For example, in some embodiments, discovered mirages (e.g., potential mirages) may be ranked based on rating scores, etc. Further, for example, in some embodiments, one or more descriptions may be ranked based on user preferences, organizational preferences, etc.

[0146] Similar to filters, in some embodiments, the various conditions, rules, metrics, or features used to explain the ranking may be based on a variety of sources, including organizational preferences, user preferences, localization, user input, other configuration information, etc.

[0147] At block 1010, in one or more of the various embodiments, the assessment engine may be arranged to generate a mirage assessment report. In one or more of the various embodiments, the mirage assessment report may include a variety of information, including a ranked list of potential or discovered mirages, a description of one or more characteristics of the various mirages, a visualization showing how the mirages appear in different contexts or views, etc.

[0148] Control may then be returned to the calling process in one or more of the various embodiments.

[0149] 11 illustrates a flowchart of a process 1100 for determining a visualization, which may include one or more visualization mirages, according to one or more of various embodiments. After a start block, in one or more of various embodiments, an evaluation engine may be provided with an original visualization, at block 1102. Thus, in one or more of various embodiments, the evaluation engine may be provided with a visualization that has been generated based on a visualization model, one or more data models, one or more data sources, etc.

[0150] In one or more of the various embodiments, the evaluation engine may be provided with models, data, or information that may be used to generate the visualization.

[0151] At block 1104, in one or more of various embodiments, the evaluation engine may be arranged to determine a next evaluation model. As mentioned above, in some embodiments, there may be several different evaluation models that may be used to find mirages in a given visualization. In some embodiments, the evaluation models may be provided in any order. However, in some embodiments, evaluation models of different types or classes may be grouped such that they may be used together or approximately in time. For example, in some embodiments, there may be one or more classes of evaluation models that require one or more of the same processing actions as part of the determination of the visualized mirage. Thus, in some embodiments, evaluation models that may rely on some of the same data processing may be executed together or approximately in time.

[0152] Similarly, in some embodiments, two or more valuation models may be directed to the same or similar types of mirages. Thus, in some embodiments, when valuation models are directed to the same type of mirage, models that consume fewer resources (or are faster) are ordered to be applied before models that consume more resources are executed.

[0153] Also, in some embodiments, the rating engine may be arranged to order the sequence rating models based on one or more recent rules. In some embodiments, the rating engine may be arranged to track or record mirage history for individual authors, data sources, data models, organizations, visualization classes, etc. Thus, in one or more of various embodiments, the mirage history may be taken into account to order or sort the sequence rating models. For example, in some embodiments, if a particular visualization author has a history of introducing mirages of a particular class, then the rating models used to discover such mirages may be run first.

[0154] However, in some embodiments the rating engine may be arranged to use rules, instructions, etc. via configuration information that may be used to take into account local circumstances or conditions.

[0155] At block 1106, in one or more of various embodiments, the evaluation engine may be arranged to render one or more test visualizations based on the current evaluation model. As described above, the evaluation model may include one or more instructions for generating the test visualizations based on the same data used to generate the original visualization. For example, in some embodiments, the evaluation model may define rules for generating modified visualization models that may be used to generate the test visualizations, such as swapping axes, adjusting baseline values, sampling data values ​​used to aggregate, changing the plotting order of different marks, and the like, or combinations thereof. In some embodiments, additional fields from the data source may be included or substituted to generate the test visualizations.

[0156] At block 1108, in one or more of various embodiments, the evaluation engine may be arranged to compare the original visualization with the test visualization to determine whether there may be a mirage included in the original visualization.

[0157] In one or more of various embodiments, the particular comparison and associated criteria may depend on a given valuation model. Thus, in some embodiments, as described above, the valuation model may include or reference one or more rules, instructions, conditions, parameters, etc. that may determine how the comparison may be performed and how the results of the comparison may be interpreted.

[0158] In one or more of various embodiments, the comparison may include generating or rendering a flattened image (e.g., a bitmap, etc.) that allows various machine vision or statistical analyses to be performed to compare the original visualization with the test visualization.

[0159] In some embodiments, the evaluation engine may be arranged to perform conventional machine vision methods to quantify various differences between the original visualization or the test visualization. For example, in some embodiments, if the visualization is robust (e.g., a visualization that is free of mirage), minor changes to the data or visualization model should be expected to result in minor changes to the appearance of the visualization. Similarly, in many cases, insubstantial changes to the appearance of the visualization (e.g., colors, plot order, orientation, etc.) should produce minor changes to the appearance of the visualization relative to an otherwise robust visualization.

[0160] At decision block 1110, in one or more of the various embodiments, if the deviation between the original visualization and the one or more test visualizations exceeds a defined threshold, control may flow to block 1112, otherwise control may flow to decision block 1114. As described above, the evaluation model may be arranged to generate an evaluation score, which may represent a probability or confidence score that a particular mirage was detected.

[0161] In one or more of various embodiments, at block 1112, the rating engine may be arranged to associate rating information, including a rating score, with the original visualization. Thus, in some embodiments, the associated rating information may be provided to a user or creator of the rated visualization. For example, if a mirage can be detected, a report may be generated or provided to one or more users indicating the type of mirage, the data or change in appearance that enabled the discovery of the mirage, etc.

[0162] In one or more of various embodiments, the evaluation model may be arranged to provide an evaluation score or the like (e.g., a confidence score) that may indicate the strength or quality of the judgment. Thus, in one or more of various embodiments, the evaluation engine may use the evaluation score provided by the evaluation model rather than a particular bias between the original visualization and the test visualization.

[0163] In decision block 1114, in one or more of the various embodiments, if there are additional evaluation models to execute, control may loop back to block 1104, otherwise control may be returned to the calling process.

[0164] FIG. 12 illustrates a flowchart of a process 1200 for evaluating a visualization that may include one or more visualization mirages, according to one or more of various embodiments. After a start block, in one or more of various embodiments, at block 1202, an evaluation engine may be provided with one or more evaluation models and one or more visualizations. As described above, in some embodiments, one or more evaluation models may be defined for detecting visualization mirages. Thus, in some embodiments, the evaluation engine may be arranged to retrieve one or more evaluation models from a data storage device or the like. In some embodiments, the evaluation engine may be arranged to determine a particular evaluation model based on rules, conditions, etc., which may be provided via configuration information.

[0165] In one or more of various embodiments, the assessment engine may be arranged to use a pipeline of different assessment models to assess whether a visualization may be associated with one or more mirages. In some embodiments, each stage of the pipeline may be associated with a different stage of information processing that contributes to the final visualization. In some embodiments, each stage of this pipeline may allow for the introduction of a visualization mirage. Thus, in some embodiments, an assessment model may be designed or configured for each stage of the pipeline.

[0166] At block 1204, in one or more of the various embodiments, the evaluation engine may be arranged to execute one or more evaluation models that may be directed to the data curation. Thus, in one or more of the various embodiments, the evaluation models may be arranged to assess various characteristics of the curation of the data used to generate the visualization. In some embodiments, this may include assessing various characteristics of the source data on which the visualization may be based, such as missing or repeated records, outliers, spelling errors, drill-down bias, etc.

[0167] In one or more embodiments, missing or repeated records may be of concern because users may often assume that there is only one entry for each piece of data relevant to a visualization, but in some cases data entry or consolidation may result in missing or repeated values ​​that may result in inaccurate aggregations or groupings.

[0168] In one or more of various embodiments, outliers may be of interest because many forms of analysis can assume that data are of similar magnitude and generated by similar processes. Thus, in some embodiments, outliers, whether erroneous values ​​or unexpected extreme values, can have a significant impact on aggregations and undermine the assumptions behind many statistical tests and summaries.

[0169] In one or more of various embodiments, spelling errors may be of concern because, in some cases, strings of characters may be interpreted as categorical data for aggregation purposes, and thus, in some embodiments, typos or inconsistent spelling and capitalization, when so interpreted, may create false categories or remove important data from aggregate queries.

[0170] In one or more of various embodiments, drill-down bias may be of concern because users assume that the order in which data is examined will not affect their conclusions. However, in some cases, filtering for less explanatory or relevant variables first may mask the full extent of the influence of later variables. Thus, in some embodiments, this may result in insights that only address small portions of data when they fit into a larger whole.

[0171] At block 1206, in one or more of the various embodiments, the valuation engine may be arranged to execute one or more valuation models, which may be directed to data processing or data transformation.

[0172] Thus, in one or more of various embodiments, the evaluation model may be arranged to analyze the data processing for the presence of different numbers of records per group. In some aspects, these may be of interest because certain summary statistics, including aggregates, may be sensitive to sample size. However, the number of records aggregated into a single visualization mark may vary dramatically. In some cases, this discrepancy may mask sensitivity and hinder mark-by-mark comparisons. Also, in some embodiments, when combined with different levels of aggregation, it may result in counterintuitive results such as Simpson's paradox.

[0173] In one or more of the various embodiments, the evaluation model may be arranged to determine the extent to which cherry-picked data may be used in the visualization. In some embodiments, filtering and subsetting are meant to be tools that remove irrelevant data or allow the user to focus on specific areas of interest. However, if this filtering is too aggressive, or if the user focuses on individual examples rather than general trends, this cherry-picking may encourage erroneous conclusions or biased views of the relationships between variables. Thus, in some embodiments, the failure to hold the broader dataset in context may result in the Texas Sharpshooter fallacy or other forms of HARKing.

[0174] In one or more of the various embodiments, the evaluation model may be arranged to determine the extent to which the degrees of freedom may have influenced the visualization. In some embodiments, this may be of interest because the author's visualization allows a great deal of flexibility in how the user analyzes the data. In some embodiments, such "researcher's degrees of freedom" may create conclusions that may be very specific to the choices made by the analyst, or, in a malicious sense, encourage "p-hacking," where the analyst explores the parameter space to find the best support for a pre-given conclusion. In some embodiments, a related problem that may be encountered is the "multiple comparison problem," where the user makes so many choices that it is likely that at least one configuration will appear significant by chance, even if there is no strong signal in the data.

[0175] In one or more of the various embodiments, the evaluation model may be arranged to determine the degree of confounding imputation that may have affected the visualization. In some embodiments, this may be of interest because there are many strategies for dealing with missing or incomplete data, including imputing new values. Thus, in some embodiments, how values ​​are imputed and then how these imputed values ​​are visualized in the context of the rest of the data may affect how the data is perceived, which may create spurious trends or group differences that may simply be artifacts of how missing values ​​are handled prior to visualization.

[0176] In one or more of the various embodiments, the evaluation engine may be arranged to evaluate the visualization at block 1208. In one or more of the various embodiments, the evaluation model may be arranged to determine whether the appearance of the visualization may contribute to one or more visualization mirages.

[0177] In one or more of the various embodiments, the evaluation model may be arranged to assess whether the visualization includes irrational visualizations. In some embodiments, these may be of concern because users expect graphics that look like charts to be a mapping between data and images. In some cases, visualizations that are used as decoration (where the marks do not relate to the data) present non-information that may be mistaken for real information. Thus, in some cases, even if the data is accurate, additional unjustified annotations may create a misleading impression, such as embellishing uncorrelated data with false approximation lines, etc.

[0178] In one or more of the various embodiments, the evaluation model may be arranged to assess whether the visualization contains overplots. In some embodiments, this may be of interest because users typically expect to be able to clearly identify individual marks in a visualization and expect one visual mark to correspond to a single value or aggregate value. However, in some cases, overlapping marks may hide internal structure in the distribution or disguise potential data quality issues.

[0179] In one or more of the various embodiments, the evaluation model may be arranged to assess whether the visualization contains hidden uncertainties. In some embodiments, this may be of interest because visualizations that do not indicate the inclusion of uncertainty risks may give the reader the wrong impression and the potential for extreme distrust of the data if they do not realize that the information is not clearly presented. In some cases, there may be a tendency to erroneously assume that the data is of high quality or complete, even if there is no evidence of veracity.

[0180] In one or more of the various embodiments, the evaluation model may be arranged to assess whether the visualization includes manipulation of scale. In some embodiments, this may be of interest because the axes and scales of the visualization are assumed by the user to directly represent quantitative information. However, manipulation of these scales (e.g., by inverting them from their commonly assumed orientation, truncating or expanding them with respect to the range of the data, using non-linear transformations, using dual axes, etc.) may cause the viewer to misinterpret the data in the chart, for example, by exaggerating correlations, exaggerating effect sizes, misinterpreting the direction of effects, etc.

[0181] At block 1210, in one or more of the various embodiments, the evaluation engine may be arranged to evaluate a reader effect that may be associated with the visualization.

[0182] In one or more of the various embodiments, the evaluation model may be arranged to assess whether the visualization contains base-rate bias. In some embodiments, this may be of interest because users may assume that unexpected values ​​in the visualization are representative of reliable differences. However, in some cases, readers may not be aware of the associated base rate, i.e., the relative likelihood that what they see is a surprising value, or the false discovery rate of the overall analysis process.

[0183] In one or more of various embodiments, the evaluation model may be arranged to evaluate whether a visualization depends on one or more inaccessible visualizations. In some embodiments, this may be of interest because the creator of the visualization may assume that users are a homogenous group. However, the way different people read a visualization may be heterogeneous and depend on underlying perceptual abilities and cognitive background that may be overlooked by the creator of the visualization. Thus, in some aspects, insufficient attention to these differences may result in miscommunication. For example, for some embodiments, a color-blind user may interpret two colors as the same when the creator of the visualization intended them to be separate.

[0184] In one or more of the various embodiments, the evaluation model may be arranged to evaluate whether the visualization may include an anchoring effect. In some embodiments, this may be of interest because the framing of early information tends to guide subsequent judgments. In some cases, this may cause readers to place undue rhetorical weight on early observations, which may cause them to underestimate or distrust subsequent observations.

[0185] In one or more of various embodiments, the evaluation model may be arranged to assess whether the visualization may be redundant with biases in the interpretation. In some embodiments, this may be of interest because each viewer arrives at the visualization with their own preconceptions, biases, and epistemological frameworks. Thus, in some embodiments, if these biases are not carefully considered, various cognitive biases, such as the backfire effect or confirmation bias, may cause a viewer to anchor only on data (or readings of data) that support their preconceptions, reject data that are inconsistent with their views, and generally ignore a more holistic picture of the strength of the evidence.

[0186] At block 1212, in one or more of the various embodiments, the evaluation engine may be arranged to execute one or more other evaluation models that may be directed to the same or other sources of visualization mirages.

[0187] Thus, in some embodiments, the rating engine may be arranged to use one or more rules, instructions, conditions, etc., which may be provided by the configuration information, to take into account local circumstances, including obtaining or otherwise determining one or more customized or localized rating models.

[0188] Control may then be returned to the calling process in one or more of the various embodiments.

[0189] It will be understood that each block in each flowchart diagram, and combinations of blocks in each flowchart diagram, may be implemented by computer program instructions. These program instructions are provided to a processor to generate a machine such that the instructions executed on the processor create means for implementing the action specified in each flowchart block or blocks. The computer program instructions are executed by the processor to cause a series of operational steps to be performed by the processor to generate a computer-implemented process such that the instructions executed on the processor provide steps for implementing the action specified in each flowchart block or blocks. The computer program instructions may also cause at least some of the operational steps shown in each flowchart block to be executed in parallel. Furthermore, some of the steps may also be executed across two or more processors, as may occur in a multi-processor computer system. Additionally, one or more blocks or combinations of blocks in each flowchart diagram may be executed simultaneously with other blocks or combinations of blocks, or even in a different sequence than shown, without departing from the scope or spirit of the present invention.

[0190] Thus, each block in each flowchart diagram supports a combination of means for performing the specified actions, a combination of steps for performing the specified actions, and program instruction means for performing the specified actions. It will also be understood that each block in each flowchart diagram, and combinations of blocks in each flowchart diagram, may be implemented by a special purpose hardware-based system that performs the specified actions or steps, or a combination of special purpose hardware and computer instructions. The foregoing examples should not be construed as limiting or exhaustive, but are illustrative use cases to illustrate at least one implementation of various embodiments of the present invention.

[0191] Additionally, in one or more embodiments (not shown), the logic in the exemplary flow charts may be executed using an embedded logic hardware device instead of a CPU, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or the like, or a combination thereof. The embedded logic hardware device may directly execute its embedded logic to perform the actions. In one or more embodiments, a microcontroller may be arranged to execute its own embedded logic to perform the actions and access its own internal memory and its own external input and output interfaces (e.g., hardware pins or a wireless transceiver) to perform the actions, such as a system on a chip (SOC).

Claims

1. 1. A method for managing visualization of data using one or more processors executing instructions for performing actions, the actions comprising: providing one or more visualizations based on data from a data source; determining one or more valuation models based on the one or more visualizations, the one or more valuation models arranged to detect one or more mirages in the one or more visualizations, the one or more mirages being visually misleading presentations of data in the one or more visualizations; and determining one or more assessment results based on the one or more visualizations and data from the data sources using the one or more assessment models, each assessment result including an assessment score corresponding to detection of the one or more mirages; ranking the one or more evaluation results based on the evaluation score; and and providing a report comprising a ranked list of the one or more evaluation results.

2. Determining the one or more evaluation results using the one or more evaluation models includes: generating one or more test visualizations based on the one or more visualizations, each test visualization being modified based on the one or more evaluation models; comparing a difference between the one or more test visualizations and the one or more visualizations; 2. The method of claim 1, further comprising: determining a probability score corresponding to detection of the one or more mirages based on the comparison, the probability score being proportional to a magnitude of the compared differences.

3. 2. The method of claim 1 , wherein using the one or more evaluation models to determine the one or more evaluation results further comprises evaluating the data associated with the one or more visualizations to determine one or more of missing or repeated records, spelling errors, drill-down bias, a number of different records per group, or misleading data selection.

4. 2. The method of claim 1 , wherein using the one or more evaluation models to determine the one or more evaluation results further comprises evaluating the one or more visualizations to determine one or more types of mirages, including irrational visualizations, over-plotting, hidden uncertainties, or manipulation of scale.

5. Determining the one or more evaluation results using the one or more evaluation models includes: evaluating data included in the data source and associated with the one or more visualizations to be excluded from the one or more visualizations; The method of claim 1 , further comprising generating a portion of the one or more evaluation results based on the evaluation.

6. 2. The method of claim 1 , wherein providing the report further comprises generating one or more other visualizations related to one or more of the one or more valuation models, the one or more valuation results, or the one or more mirages.

7. 1. A processor-readable non-transitory storage medium comprising instructions for managing visualization, the execution of said instructions by one or more processors comprising: providing one or more visualizations based on data from the data sources; determining one or more valuation models based on the one or more visualizations, the one or more valuation models arranged to detect one or more mirages in the one or more visualizations, the one or more mirages being visually misleading presentations of data in the one or more visualizations; and determining one or more assessment results based on the one or more visualizations and data from the data sources using the one or more assessment models, each assessment result including an assessment score corresponding to the detection of one or more mirages; ranking the one or more evaluation results based on the evaluation score; and providing a report including a ranked list of the one or more evaluation results.

8. 1. A system for managing visualization, comprising:

1. A network computer, comprising: a transceiver for communicating over a network; a memory for storing at least instructions; and one or more processors executing instructions to perform actions, the actions including: providing one or more visualizations based on data from the data sources; determining one or more valuation models based on the one or more visualizations, the one or more valuation models arranged to detect one or more mirages in the one or more visualizations, the one or more mirages being visually misleading presentations of data in the one or more visualizations; and determining one or more assessment results based on the one or more visualizations and data from the data sources using the one or more assessment models, each assessment result including an assessment score corresponding to the detection of one or more mirages; ranking the one or more evaluation results based on the evaluation score; and providing a report including a ranked list of the one or more evaluation results; and A client computer, a transceiver for communicating over the network; a memory for storing at least instructions; and one or more processors executing instructions to perform actions, the actions including: and a client computer configured to receive the report.

9. Determining the one or more evaluation results using the one or more evaluation models includes: generating one or more test visualizations based on the one or more visualizations, each test visualization being modified based on the one or more evaluation models; comparing a difference between the one or more test visualizations and the one or more visualizations; 9. The system of claim 8, further comprising: determining a probability score corresponding to detection of the one or more mirages based on the comparison, the probability score being proportional to a magnitude of the compared differences.

10. 10. The system of claim 8, wherein using the one or more evaluation models to determine the one or more evaluation results further comprises evaluating the data associated with the one or more visualizations to determine one or more of missing or repeated records, spelling errors, drill-down bias, a number of different records per group, or misleading data selection.

11. 10. The system of claim 8, wherein using the one or more evaluation models to determine the one or more evaluation results further comprises evaluating the one or more visualizations to determine one or more types of mirages, including irrational visualization, over-plotting, hidden uncertainty, or manipulation of scale.

12. Determining the one or more evaluation results using the one or more evaluation models includes: evaluating data included in the data source and associated with the one or more visualizations to be excluded from the one or more visualizations; The system of claim 8 , further comprising generating a portion of the one or more assessment results based on the assessment.

13. 10. The system of claim 8, wherein providing the report further comprises generating one or more other visualizations related to one or more of the one or more valuation models, the one or more valuation results, or the one or more mirages.

14. A computer for managing the visualization, a transceiver for communicating over a network; A method and apparatus for implementing the method of the present invention comprising: a memory for storing at least instructions; and one or more processors for executing the instructions to perform actions, said actions being: providing one or more visualizations based on data from the data sources; determining one or more valuation models based on the one or more visualizations, the one or more valuation models arranged to detect one or more mirages in the one or more visualizations, the one or more mirages being visually misleading presentations of data in the one or more visualizations; and determining one or more assessment results based on the one or more visualizations and data from the data sources using the one or more assessment models, each assessment result including an assessment score corresponding to the detection of one or more mirages; ranking the one or more evaluation results based on the evaluation score; and providing a report including a ranked list of the one or more evaluation results.

15. Determining the one or more evaluation results using the one or more evaluation models includes: generating one or more test visualizations based on the one or more visualizations, each test visualization being modified based on the one or more evaluation models; comparing a difference between the one or more test visualizations and the one or more visualizations; 15. The computer of claim 14, further comprising: determining a probability score corresponding to detection of the one or more mirages based on the comparison, the probability score being proportional to a magnitude of the compared differences.

Images