Reveal the visual illusion

Abstract

Implementations relate to managing visualizations. A visualization based on data from a data source can be provided. An evaluation model based on the visualization can be provided, such that the evaluation model can detect an artifact in the visualization. The evaluation model can be used to determine evaluation results based on the visualization and the data from the data source, such that each evaluation result includes an evaluation score corresponding to the artifact detection. The evaluation results can be rank ordered based on the evaluation scores. A report including a rank ordered list of the evaluation results can be provided.

Description

[0001] Cross-reference to related applications

[0002] This application is based on a utility patent application previously filed, U.S. Provisional Patent Application No. 62 / 902,273, filed on September 18, 2019, which claims the benefit of its filing date under 35 U.S.SC §119(e) and is further incorporated herein by reference in its entirety. Technical Field

[0003] This invention relates generally to data visualization, and more specifically, but not exclusively, to the automatic identification of potentially misleading visualizations. Background Technology

[0004] Organizations are generating and collecting ever-increasing amounts of data. This data can be correlated with different parts of the organization, such as consumer activities, manufacturing activities, customer service, server logs, etc. For various reasons, such organizations may find it inconvenient to effectively utilize their massive data collections. In some cases, the sheer volume of data can make it difficult to effectively leverage the collected data to improve business practices. In other cases, organizations use various tools to generate visualizations of some or all of their data. Using visualizations to represent this data can help organizations improve their understanding of critical business operations and help them monitor key performance indicators. However, in some cases, visualizations may include mirages that can mislead viewers or the creators of the visualization, or even appear perfectly normal. In some cases, identifying or otherwise analyzing the presence, origin, or cause of such mirages may require an over-interpretation of the underlying data used to generate the visualization. Disadvantageously, this may require organizations to instruct skilled or specialized data analysts to review the visualizations, and the data may help identify mirages that could mislead the audience. Furthermore, in some cases, even if users have the skills or technical background to perform their own mirage analysis, users reviewing or validating a visualization may not have access to the underlying data. Therefore, this invention is proposed in relation to these and other considerations. Attached Figure Description

[0005] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following accompanying drawings. In the drawings, unless otherwise stated, the same reference numerals refer to the same parts throughout the various figures. For a better understanding of the described innovations, reference will be made to the following detailed description of various embodiments, which will be read in conjunction with the accompanying drawings, wherein:

[0006] Figure 1 The system environment that can implement various implementation schemes is shown;

[0007] Figure 2 A schematic implementation scheme for the client computer is shown;

[0008] Figure 3 A schematic implementation of a network computer is shown;

[0009] Figure 4 The logical architecture of a system for revealing a visual illusion is shown according to one or more of various implementation schemes;

[0010] Figure 5A A representation of a portion of a visualization for revealing a visual illusion, according to one or more of various embodiments, is shown;

[0011] Figure 5B A representation of a portion of a visualization for revealing a visual illusion, according to one or more of various embodiments, is shown;

[0012] Figure 6A This illustrates a portion of the visualization, which includes a bar chart comparing house prices;

[0013] Figure 6B This illustrates a portion of the visualization, which includes a bar chart comparing house prices;

[0014] Figure 7 A logical representation of a portion of an evaluation system according to one or more of various implementation schemes is shown;

[0015] Figure 8 A logical representation of a portion of a user interface for revealing a visual illusion, according to one or more of various implementations, is shown;

[0016] Figure 9 An overview flowchart of a process for revealing a visual illusion, according to one or more of various implementation schemes, is shown;

[0017] Figure 10 A flowchart is shown for evaluating visualizations to discover visual illusions, according to one or more of various implementation schemes;

[0018] Figure 11 A flowchart is shown, according to one or more of various embodiments, of a process for determining a visualization that may include one or more visual illusions; and

[0019] Figure 12 A flowchart is shown for evaluating a visualization process that may include one or more visual illusions, according to one or more of various implementation schemes. Detailed Implementation

[0020] Various embodiments will now be described more fully with reference to the accompanying drawings, which form part of the invention and illustrate specific exemplary embodiments in which the invention may be practiced. However, embodiments may be implemented in many different forms and should not be construed as limited to the embodiments described herein; rather, these embodiments are provided so that the disclosure will be thorough and complete and will fully convey the scope of the embodiments to those skilled in the art. Among other things, the various embodiments may be methods, systems, media, or devices. Thus, the various embodiments may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Therefore, the detailed description below should not be construed as limiting.

[0021] Throughout the specification and claims, unless the context clearly specifies otherwise, the following terms shall have the meaning explicitly associated herein. The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment (although it may). Furthermore, the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment (although it may). Therefore, as described below, various embodiments can be readily combined without departing from the scope or spirit of the invention.

[0022] Furthermore, as used herein, the term “or” is an inclusive “or” operator and is equivalent to the term “and / or”, unless the context explicitly specifies otherwise. The term “based on” is not exclusive and allows for basing on other factors not described, unless the context explicitly specifies otherwise. Additionally, throughout the specification, the meanings of “a,” “an,” and “the” include plural references. The meaning of “in” includes both “in” and “on”.

[0023] For example, in the implementation plan, the following terms are also used here according to their respective meanings, unless the context clearly indicates otherwise.

[0024] As used in this article, the term "engine" refers to the logic contained within hardware or software instructions that can be written in programming languages ​​such as C, C++, Objective-C, COBOL, and Java. TM , PHP, Perl, JavaScript, Ruby, VBScript, Microsoft.NET TM Languages, such as C#, are used. Engines can be compiled into executable programs or written in interpreted programming languages. Software engines can be invoked from other engines or from themselves. The engine described in this document refers to one or more logical modules that can be merged with other engines or applications, or can be divided into sub-engines. Engines can be stored on non-transitory computer-readable media or computer storage devices, and can be stored on and executed by one or more general-purpose computers, thereby creating a dedicated computer configured to provide the engine.

[0025] As used herein, the term "data source" refers to a database, application, service, file system, etc., that stores or provides information to an organization. Examples of data sources can include RDBMS databases, graph databases, spreadsheets, file systems, document management systems, local or remote data streams, etc. In some cases, a data source is organized around a structure of one or more tables or similar tables. In other cases, a data source is organized as a chart or similar chart structure.

[0026] As used herein, the term "data model" refers to one or more data structures that provide a representation of one or more parts of an underlying data source. In some cases, a data model can provide a view of the data source for a specific application. A data model can be thought of as a view or interface to the underlying data source. In some cases, a data model can be directly mapped to the data source (e.g., practically a logical pass-through). Furthermore, in some cases, a data model can be viewed as an interface to the data source. Data models enable organizations to organize or present information from data sources in a more convenient, meaningful (e.g., easier to reason about), secure, and other ways.

[0027] As used herein, the term "data object" refers to one or more data structures that comprise a data model. In some cases, a data object can be considered part of a data model. A data object can represent a single instance of an item or a class or kind of item.

[0028] As used herein, the term "panel" refers to an area within a graphical user interface (GUI) that has a defined geometry (e.g., x, y, z order) within the GUI. Panels can be arranged to display information to the user or host one or more interactive controls. Configuration information, including dynamic rules, can be used to define the geometry or style associated with a panel. Furthermore, in certain situations, users can perform actions on one or more panels, such as moving, showing, hiding, resizing, reordering, etc.

[0029] For users of this article, "visualization model" refers to one or more data structures representing one or more representations of a data model (which can be adapted for use in visualizations displayed on one or more hardware displays). Visualization models may have defined style or user interface characteristics that are available to non-authoring users.

[0030] As used herein, the term "display object" refers to one or more data structures that comprise a visualization model. In some cases, a display object may be considered part of the visualization model. A display object can represent a single instance of an item that can be displayed in a visualization, or an entire class or category of items. In some implementations, display objects may be considered or referred to as views because they provide a view of certain parts of the data model.

[0031] As used in this article, the term "visual illusion" or "illusion" refers to a visually misleading presentation of data in a visualization. Furthermore, a visual illusion can be any visualization in which a cursory reading of the visualization seems to support a specific message derived from the data, but a more careful re-examination of the visualization, the supporting data, or the analytical process may invalidate or raise significant doubts about this support.

[0032] As used herein, the term “configuration information” refers to information that may include rule-based policies, pattern matching, scripts (e.g., computer-readable instructions), etc., which may be available from a variety of sources, including configuration files, databases, user input, built-in defaults, etc., or a combination thereof.

[0033] The following is a brief description of embodiments of the invention to provide a basic understanding of some aspects of the invention. This brief description is not intended as a broad overview. It is not intended to identify key or critical elements, nor to define or otherwise narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that follows.

[0034] In short, various implementation schemes involve using one or more processors (executing one or more instructions to perform the operations described herein) to manage data visualization. In one or more of these implementation schemes, one or more visualizations based on data from a data source can be provided.

[0035] In one or more of the various implementation schemes, one or more evaluation models based on one or more visualizations can be provided, such that the one or more evaluation models can be arranged to detect one or more illusions in one or more visualizations, and such that the one or more illusions can be visually misleading representations of data in one or more visualizations.

[0036] In one or more of the various implementation schemes, one or more evaluation models can be used to determine one or more evaluation results based on one or more visualizations and data from a data source, such that each evaluation result includes an evaluation score corresponding to the detection of one or more phantoms.

[0037] In one or more of the various implementation schemes, using one or more evaluation models to determine one or more evaluation results may include: generating one or more test visualizations based on one or more visualizations, such that each test visualization is modified based on one or more evaluation models; comparing the differences between one or more test visualizations and one or more visualizations; and determining a probability score for the detection corresponding to one or more phantoms based on the comparison, such that the value of the probability is proportional to the magnitude of the difference in the comparison.

[0038] In one or more of the various implementation schemes, using one or more evaluation models to determine one or more evaluation results may include: evaluating data associated with one or more visualizations to determine one or more of the following: missing or duplicate records, spelling errors, drill-down bias, differing number of records by group, or misleading data selection.

[0039] In one or more of the various implementation schemes, using one or more evaluation models to determine one or more evaluation results may include: evaluating one or more visualizations to determine one or more types of illusions, including one or more of discontinuous visualizations, overdrawing, hidden uncertainties, and scale manipulation.

[0040] In one or more of the various implementation schemes, using one or more evaluation models to determine one or more evaluation results may include: evaluating data associated with one or more visualizations (including in the data source and omitted from one or more visualizations); and generating a portion of one or more evaluation results based on the evaluation.

[0041] In one or more of the various implementation schemes, one or more evaluation results may be ranked based on evaluation scores.

[0042] In one or more of the various implementations, a report may be provided that includes a ranking list of one or more evaluation results. In one or more of the various implementations, providing the report may include generating one or more other visualizations that can be associated with one or more of the evaluation models, one or more evaluation results, or one or more illusions.

[0043] Illustrated operating environment

[0044] Figure 1The diagram illustrates components of one embodiment in an environment in which embodiments of the invention can be practiced. Implementing the invention may not require all components, and the arrangement and type of components can be varied without departing from the spirit or scope of the invention. As shown in the figure, Figure 1 System 100 includes a local area network (LAN) / wide area network (WAN) 110, a wireless network 108, client computers 102-105, a visualization server computer 116, a data source server computer 118, etc.

[0045] The following is combined Figure 2 At least one embodiment of client computers 102-105 is described in more detail. In one embodiment, at least some of client computers 102-105 can operate on one or more wired or wireless networks, such as network 108 or 110. Typically, client computers 102-105 can include virtually any computer capable of communicating over a network to send and receive information, perform various online activities, offline actions, etc. In one embodiment, one or more client computers 102-105 can be configured to operate within an enterprise or other entity to perform various services for that enterprise or other entity. For example, client computers 102-105 can be configured to operate as a web server, firewall, client application, media player, mobile phone, game console, desktop computer, etc. However, client computers 102-105 are not limited to these services and can also be used, for example, for end-user computing in other embodiments. It should be recognized that more or less client computers (such as...) Figure 1 (As shown) can be included in systems such as those described herein, and therefore the implementation is not limited by the number or type of client computers used.

[0046] Computers that can operate as client computers 102 may include computers typically connected via wired or wireless communication media, such as personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, network PCs, etc. In some embodiments, client computers 102-105 may include virtually any portable computer capable of connecting to another computer and receiving information, such as laptop computers 103, mobile computers 104, tablet computers 105, etc. However, portable computers are not limited to this and may also include other portable computers, such as cellular phones, display pagers, radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), handheld computers, wearable computers, integrated devices combining one or more of the aforementioned computers, etc. Therefore, client computers 102-105 are generally wide-ranging in terms of capabilities and features. Furthermore, client computers 102-105 can access a variety of computing applications, including browsers or other web-based applications.

[0047] Web-enabled client computers may include browser applications configured to send requests and receive responses over the Web. Browser applications can be configured to receive and display graphics, text, multimedia, etc., using virtually any Web-based language. In one implementation, the browser application is enabled to display and send messages using JavaScript, Hypertext Markup Language (HTML), Extensible Markup Language (XML), JavaScript Object Notation (JSON), Cascading Style Sheets (CSS), and combinations thereof. In one implementation, a user of the client computer can use the browser application to perform various activities over the network (online). However, another application may also be used to perform various online activities.

[0048] Client computers 102-105 may also include at least one other client application configured to receive or send content between other computers. The client application may include the ability to send or receive content, etc. The client application may also provide information identifying itself, including type, function, name, etc. In one embodiment, client computers 102-105 may uniquely identify themselves through any of a variety of mechanisms, including Internet Protocol (IP) address, telephone number, mobile identification number (MIN), electronic serial number (ESN), client certificate, or other device identifier. Such information may be provided in one or more network packets, etc., and sent between other client computers, visualization server computer 116, data source server computer 118, or other computers.

[0049] Client computers 102-105 can also be configured to include client applications that enable end users to log in to an end-user account, which can be managed by another computer, such as visualization server computer 116, data source server computer 118, etc. In a non-limiting example, such an end-user account can be configured to enable the end user to manage one or more online activities, including, in a non-limiting example, project management, software development, system management, configuration management, search activities, social networking activities, browsing various websites, and communicating with other users. Furthermore, the client computers can be arranged to enable users to display reports, interactive user interfaces, or results provided by visualization server computer 116 and data source server computer 118.

[0050] Wireless network 108 is configured to couple client computers 103-105 and their components to network 110. Wireless network 108 may include any of a variety of wireless subnetworks, which may further cover independent ad-hoc networks, etc., to provide infrastructure-oriented connectivity for client computers 103-105. Such subnetworks may include mesh networks, wireless LAN (WLAN) networks, cellular networks, etc. In one embodiment, the system may include more than one wireless network.

[0051] The wireless network 108 may also include autonomous systems such as terminals, gateways, and routers connected via wireless radio links. These connectors can be configured to move freely and randomly and organize themselves arbitrarily, allowing the topology of the wireless network 108 to change rapidly.

[0052] Wireless network 108 may further employ multiple access technologies, including second (2G), third (3G), fourth (4G), and fifth (5G) generation radio access for cellular systems, WLAN, wireless router (WR) mesh networks, etc. Access technologies such as 2G, 3G, 4G, 5G, and future access networks enable mobile computers, such as client computers 103-105 with varying degrees of mobility, to achieve wide-area coverage. In a non-limiting example, wireless network 108 may achieve radio connectivity via radio network access, such as Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), etc. In essence, wireless network 108 may include virtually any wireless communication mechanism through which information can be transmitted between client computers 103-105 and another computer, network, cloud-based network, cloud instance, etc.

[0053] Network 110 is configured to couple network computers to other computers, including visualization server computer 116, data source server computer 118, client computer 102, and client computers 103-105, via wireless network 108, etc. Network 110 is capable of using any form of computer-readable medium to transmit information from one electronic device to another. Furthermore, in addition to local area networks (LANs), wide area networks (WANs), direct connections such as via universal serial bus (USB) ports, Ethernet ports, other forms of computer-readable media, or any combination thereof, network 110 may also include the Internet. On a set of interconnected LANs, including those based on different architectures and protocols, routers act as links between LANs, enabling messages to be sent from one LAN to another. Furthermore, communication links within a LAN typically include twisted-pair cables or coaxial cables, while communication links between networks may utilize analog telephone lines, all or part of dedicated digital lines including T1, T2, T3, and T4, or other carrier mechanisms, including, for example, electronic carriers, Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), wireless links including satellite links, or other communication links known to those skilled in the art. Furthermore, the communication link may employ any of a variety of digital signaling technologies, including but not limited to DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, etc. Additionally, remote computers and other related electronic devices can be remotely connected to the LAN or WAN via modems and temporary telephone links. In one implementation, network 110 may be configured to transmit Internet Protocol (IP) information.

[0054] Furthermore, communication media typically contain computer-readable instructions, data structures, program modules, or other transmission mechanisms, and include any medium that transmits information neither instantaneously nor instantaneously. For example, communication media include wired media such as twisted-pair cables, coaxial cables, optical fibers, waveguides, and other wired media, as well as wireless media such as acoustic, RF, infrared, and other wireless media.

[0055] In addition, the following combination Figure 3 A more detailed description of one implementation of the visualization server computer 116 and the data source server computer 118 is provided. Although Figure 1Visualization server computer 116, data source server computer 118, etc., are shown as a single computer, but innovations or implementations are not limited to this. For example, one or more functions of visualization server computer 116, data source server computer 118, etc., can be distributed across one or more different network computers. Furthermore, in one or more implementations, visualization server computer 116, data source server computer 118, etc., can be implemented using multiple network computers. Additionally, in one or more of the various implementations, visualization server computer 116, data source server computer 118, etc., can be implemented using one or more cloud instances in one or more cloud networks. Therefore, these innovations and implementations should not be construed as being limited to a single environment, and other configurations and architectures are also contemplated.

[0056] Explanatory client computer

[0057] Figure 2 An embodiment of client computer 200 is shown, which may include more or fewer components than those shown. Client computer 200 may represent, for example... Figure 1 One or more implementations of the mobile computer or client computer shown.

[0058] Client computer 200 may include processor 202 communicating with memory 204 via bus 228. Client computer 200 may also include power supply 230, network interface 232, audio interface 256, display 250, keyboard 252, illuminator 254, video interface 242, input / output interface 238, haptic interface 264, Global Positioning System (GPS) receiver 258, open gesture interface 260, temperature interface 262, camera 240, projector 246, pointing device interface 266, processor-readable fixed storage device 234, and processor-readable removable storage device 236. Client computer 200 may optionally communicate with a base station (not shown) or directly with another computer. And in one embodiment, although not shown, a gyroscope may be used within client computer 200 to measure or maintain the orientation of client computer 200.

[0059] Power supply 230 can provide power to client computer 200. Rechargeable or non-rechargeable batteries can be used to provide power. Power can also be provided by an external power source, such as an AC adapter or a power docking station to replenish or recharge the battery.

[0060] Network interface 232 includes circuitry for coupling client computer 200 to one or more networks and is configured to be used with one or more communication protocols and technologies, including but not limited to protocols and technologies implementing any part of the OSI model for mobile communications (GSM), CDMA, Time Division Multiple Access (TDMA), UDP, TCP / IP, SMS, MMS, GPRS, WAP, UWB, WiMax, SIP / RTP, GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or any of various other wireless communication protocols. Network interface 232 is sometimes referred to as a transceiver, transceiver device, or network interface card (NIC).

[0061] Audio interface 256 can be configured to generate and receive audio signals, such as human voice. For example, audio interface 256 can be coupled to a speaker and a microphone (not shown) to enable telecommunication communication with others or to generate audio confirmations for certain actions. The microphone in audio interface 256 can also be used to input to or control client computer 200, for example, using voice recognition, sound-based touch detection, etc.

[0062] Display 250 may be a liquid crystal display (LCD), gas plasma, electronic ink, light-emitting diode (LED), organic LED (OLED), or any other type of light-reflecting or light-transmitting display that can be used with a computer. Display 250 may also include a touch interface 244, which is arranged to receive input from a digital object such as a stylus or a human hand, and may use resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other technologies to sense touch or gestures.

[0063] Projector 246 can be a remote handheld projector or an integrated projector, capable of projecting images onto a remote wall or any other reflective object such as a remote screen.

[0064] Video interface 242 can be configured to capture video images, such as still photographs, video clips, infrared video, etc. For example, video interface 242 can be coupled to a digital camera, a network camera, etc. Video interface 242 may include a lens, an image sensor, and other electronic equipment. The image sensor may include a complementary metal-oxide-semiconductor (CMOS) integrated circuit, a charge-coupled device (CCD), or any other integrated circuit used for sensing light.

[0065] Keyboard 252 may include any input device arranged to receive input from a user. For example, keyboard 252 may include a button numeric dialer or a keypad. Keyboard 252 may also include command buttons associated with selecting and sending images.

[0066] The illuminator 254 can provide status indication or provide light. The illuminator 254 can remain active for a specific period of time or in response to an event message. For example, when the illuminator 254 is active, it can backlight the buttons on the keyboard 252 and remain on when the client computer is powered. Furthermore, when performing specific actions, the illuminator 254 can backlight these buttons in various modes, such as dialing another client computer. The illuminator 254 can also illuminate a light source positioned within the transparent or translucent casing of the client computer in response to an action.

[0067] In addition, the client computer 200 may also include a Hardware Security Module (HSM) 268 for providing additional tamper-proof safeguards for generating, storing, or using secure / cryptographic information such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some implementations, the Hardware Security Module may be used to support one or more standard Public Key Infrastructure (PKI) systems and may be used to generate, manage, or store key pairs, etc. In some implementations, the HSM 268 may be a standalone computer; in other cases, the HSM 268 may be configured as a hardware card that can be added to the client computer.

[0068] Client computer 200 may also include an input / output interface 238 for communicating with external peripheral devices or other computers such as other client computers and network computers. Peripheral devices may include audio headsets, virtual reality headsets, display glasses, remote speaker systems, remote speaker and microphone systems, etc. Input / output interface 238 may utilize one or more technologies, such as Universal Serial Bus (USB), infrared, WiFi, WiMax, Bluetooth™, etc.

[0069] The input / output interface 238 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation scales, etc.). The sensors may be one or more hardware sensors that collect or measure data external to the client computer 200.

[0070] Haptic interface 264 can be configured to provide haptic feedback to a user of the client computer. For example, haptic interface 264 can be used to vibrate client computer 200 in a specific manner when another user of the computer is making a call. Temperature interface 262 can be used to provide a temperature measurement input or a temperature change output to a user of client computer 200. Open gesture interface 260 can sense the physical gestures of the user of client computer 200, for example, by using a single or stereo camera, radar, a gyroscope sensor within the computer held or worn by the user, etc. Camera 240 can be used to track the physical eye movements of the user of client computer 200.

[0071] GPS transceiver 258 can determine the physical coordinates of client computer 200 on the Earth's surface, typically outputting the location as latitude and longitude values. GPS transceiver 258 can also use other geolocation mechanisms, including but not limited to triangulation, assisted GPS (AGPS), enhanced time difference of observation (E-OTD), cell identifier (CI), service area identifier (SAI), enhanced timing advance (ETA), base station subsystem (BSS), etc., to further determine the physical location of client computer 200 on the Earth's surface. It should be understood that GPS transceiver 258 can determine the physical location of client computer 200 under different conditions. However, in one or more embodiments, client computer 200 may be provided with additional information through other components that can be used to determine the physical location of the client computer, including, for example, media access control (MAC) address, IP address, etc.

[0072] In at least one of the various embodiments, applications such as operating system 206, other client applications 224, web browser 226, etc., can be configured to use geolocation information to select one or more localization features, such as time zone, language, currency, calendar format, etc. Localization functionality can be used for display objects, data models, data objects, user interfaces, reports, and internal processes or databases. In at least one of the various embodiments, the geolocation information used to select the location information can be provided by GPS 258. Furthermore, in some embodiments, the geolocation information may include information provided via a network such as wireless network 108 or network 110 using one or more geolocation protocols.

[0073] The human-machine interface (HMI) component can be a peripheral device physically separate from the client computer 200, allowing remote input or output to the client computer 200. For example, information routed via an HMI component such as display 250 or keyboard 252 as described herein can alternatively be routed via network interface 232 to a suitable HMI component located remotely. Examples of remote HMI peripheral components include, but are not limited to, audio devices, pointing devices, keypads, displays, cameras, projectors, etc. These peripheral components can communicate via micronetworks such as Bluetooth™, ZigBee™, etc. A non-limiting example of a client computer having such a peripheral HMI component is a wearable computer, which may include a remote microprojector and one or more cameras, the cameras communicating remotely with a separately positioned client computer to sense gestures by the user toward a portion of an image projected by the microprojector onto a reflective surface such as a wall or the user's hand.

[0074] The client computer may include a web browser application 226 configured to receive and send web pages, web-based messages, graphics, text, multimedia, etc. The browser application on the client computer can use virtually any programming language, including Wireless Application Protocol Messages (WAP), etc. In one or more embodiments, the browser application is enabled to use Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), HTML5, etc.

[0075] Memory 204 may include RAM, ROM, or other types of memory. Memory 204 illustrates an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 204 may store a BIOS 208 for controlling the low-level operations of client computer 200. Memory may also store an operating system 206 for controlling the operation of client computer 200. It should be understood that this component may include a general-purpose operating system, such as a version of UNIX, or LINUX™, or a dedicated client computer communication operating system, such as Windows Phone™, or... Operating system. An operating system may include a Java Virtual Machine module or an interface to a Java Virtual Machine module, which enables Java applications to control hardware components or the operation of the operating system.

[0076] Memory 204 may also include one or more data memories 210, which the client computer 200 may use to store application 220 or other data. For example, data memories 210 may also be used to store information describing various capabilities of the client computer 200. The information can then be provided to another device or computer based on any of a variety of methods, including sending it as part of a header during communication, sending it upon request, etc. Data memories 210 may also be used to store social network information, including address books, friend lists, aliases, user profile information, etc. Data memories 210 may further include program code, data, algorithms, etc., for use by a processor such as processor 202 to take and perform actions. In one embodiment, at least some of the data memories 210 may also be stored on another component of the client computer 200, including but not limited to a non-transitory processor-readable removable storage device 236, a processor-readable fixed storage device 234, or even outside the client computer.

[0077] Application 220 may include computer-executable instructions that, when executed by client computer 200, send, receive, or otherwise process instructions and data. Application 220 may include, for example, a client visualization engine 222, other client applications 224, a web browser 226, etc. Client computers may be configured to exchange communication with one or more servers.

[0078] Other examples of applications include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VoIP) applications, contact managers, task managers, code converters, database programs, word processors, security applications, spreadsheet programs, games, search programs, visualization applications, and more.

[0079] Additionally, in one or more embodiments (not shown in the figures), the client computer 200 may include embedded logic hardware devices instead of a CPU, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable array logic (PALs), and combinations thereof. The embedded logic hardware devices can directly execute their embedded logic to perform actions. Furthermore, in one or more embodiments (not shown in the figures), the client computer 200 may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, the one or more microcontrollers can directly execute their own embedded logic to perform actions and access their own internal memory and their own external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as a system-on-a-chip (SoC).

[0080] Explanatory Network Computer

[0081] Figure 3 An embodiment of a network computer 300 is shown, which may be included in a system implementing one or more of various embodiments. The network computer 300 may include more than Figure 3 The components shown may include more or fewer components. However, the components shown are sufficient to disclose illustrative embodiments for practicing these innovations. The network computer 300 may represent, for example... Figure 1 An implementation of at least one of the visualization server computer 116, data source server computer 118, etc.

[0082] A network computer, such as network computer 300, may include a processor 302 that can communicate with memory 304 via bus 328. In some embodiments, processor 302 may consist of one or more hardware processors or one or more processor cores. In some cases, one or more of the processors may be special-purpose processors designed to perform one or more specific actions, such as those described herein. Network computer 300 also includes a power supply 330, a network interface 332, an audio interface 356, a display 350, a keyboard 352, an input / output interface 338, a processor-readable fixed storage device 334, and a processor-readable removable storage device 336. Power supply 330 provides power to network computer 300.

[0083] Network interface 332 includes circuitry for coupling network computer 300 to one or more networks and is configured for one or more communication protocols and technologies, including but not limited to protocols and technologies implementing any part of the Open Systems Interconnection (OSI) model, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), Short Message Service (SMS), Multimedia Message Service (MMS), General Packet Radio Service (GPRS), WAP, Ultra Wideband (UWB), IEEE 802.16, Global System for Microwave Access Interoperability (WiMax), Session Initiation Protocol / Real-Time Transport Protocol (SIP / RTP), or various other wired and wireless communication protocols. Network interface 332 is sometimes referred to as a transceiver, transceiver device, or network interface card (NIC). Network computer 300 may optionally communicate with a base station (not shown) or directly with another computer.

[0084] Audio interface 356 is configured to generate and receive audio signals, such as human voice. For example, audio interface 356 may be coupled to a speaker and microphone (not shown) to enable communication with others or to generate audio confirmations for certain actions. The microphone in audio interface 356 may also be used, for example, for input or control of network computer 300 using voice recognition.

[0085] Display 350 may be a liquid crystal display (LCD), gas plasma, electronic ink, light-emitting diode (LED), organic LED (OLED), or any other type of light-reflecting or light-transmitting display that can be used with a computer. In some embodiments, display 350 may be a handheld projector or micro-projector capable of projecting images onto a wall or other object.

[0086] Network computer 300 may also include components for communication with... Figure 3 Input / output interface 338 for communication with external devices or computers (not shown). Input / output interface 338 may utilize one or more wired or wireless communication technologies, such as USB™, Firewire™, WiFi, WiMax, Thunderbolt™, infrared, Bluetooth™, ZigBee™, serial port, parallel port, etc.

[0087] In addition, the input / output interface 338 may also include one or more sensors for determining geolocation information (e.g., GPS), monitoring power conditions (e.g., voltage sensors, current sensors, frequency sensors, etc.), monitoring weather (e.g., thermostats, barometers, anemometers, humidity detectors, precipitation scales, etc.), etc. The sensors may be one or more hardware sensors that collect or measure data external to the network computer 300. The human-machine interface components may be physically separate from the network computer 300, allowing remote input or output to the network computer 300. For example, information routed via a human-machine interface component such as display 350 or keyboard 352 as described herein may alternatively be routed via network interface 332 to an appropriate human-machine interface component located elsewhere on the network. Human-machine interface components include any components that allow the computer to receive input from or send output to a human user of the computer. Therefore, pointing devices such as mice, styluses, trackballs, etc., can communicate via pointing device interface 358 to receive user input.

[0088] GPS transceiver 340 can determine the physical coordinates of network computer 300 on the Earth's surface, typically outputting the location as latitude and longitude values. GPS transceiver 340 can also use other geolocation mechanisms, including but not limited to triangulation, assisted GPS (AGPS), enhanced time difference of observation (E-OTD), cell identifier (CI), service area identifier (SAI), enhanced timing advance (ETA), base station subsystem (BSS), etc., to further determine the physical location of network computer 300 on the Earth's surface. It should be understood that GPS transceiver 340 can determine the physical location of network computer 300 under different conditions. However, in one or more embodiments, network computer 300 may provide additional information, such as media access control (MAC) address, IP address, etc., that can be used to determine the physical location of client computers through other components.

[0089] In at least one of the various embodiments, applications such as operating system 306, evaluation engine 322, visualization engine 324, modeling engine 326, and other applications 329 may be configured to utilize geographic location information to select one or more location features, such as time zone, language, currency, currency format, calendar format, etc. Localization functionality may be used in user interfaces, dashboards, visualizations, reports, and internal processes or databases. In at least one of the various embodiments, the geographic location information used to select the location information may be provided by GPS 340. Furthermore, in some embodiments, the geographic location information may include information provided via a network such as wireless network 108 or network 111 using one or more geolocation protocols.

[0090] Memory 304 may include random access memory (RAM), read-only memory (ROM), or other types of memory. Memory 304 illustrates an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 304 stores a basic input / output system (BIOS) 308 for controlling the low-level operations of the network computer 300. Memory also stores an operating system 306 for controlling the operation of the network computer 300. It should be understood that this component may include a general-purpose operating system, such as UNIX or LINUX. TM Versions, or proprietary operating systems, such as those from Microsoft Corporation. Operating system, or Apple Corporation's Operating system. This operating system may include one or more virtual machine modules, or interfaces with one or more virtual machine modules; for example, a Java Virtual Machine module that allows Java applications to control hardware components or the operation of the operating system. Similarly, other runtime environments may also be included.

[0091] Memory 304 may further include one or more data memories 310, which may be used by network computer 300 to store application program 320 or other data. For example, data memory 310 may also be used to store information describing various capabilities of network computer 300. The information can then be provided to another device or computer based on any of a variety of methods, including sending it as part of a header during communication, sending it on request, etc. Data memory 310 may also be used to store social network information including address books, friend lists, aliases, user profile information, etc. Data memory 310 may further include program code, data, algorithms, etc., for use by a processor such as processor 302 to perform and execute actions such as those described below. In one embodiment, at least some of the data memories 310 may also be stored on another component of network computer 300, including but not limited to non-transitory media within processor-readable removable storage device 336, processor-readable fixed storage device 334, or any other computer-readable storage device within or even outside network computer 300. The data storage device 310 may include, for example, a data model 314, a data source 316, a visualization model 318, an evaluation model 319, etc.

[0092] Application 320 may include computer-executable instructions that, when executed by network computer 300, send, receive, or otherwise process messages (e.g., SMS, Multimedia Messaging Service (MMS), Instant Messaging (IM), email, or other messages), audio, video, and enable telecommunications communication with another user on another mobile computer. Other examples of applications include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VoIP) applications, contact managers, task managers, code converters, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and so on. Application 320 may include evaluation engines 322, visualization engines 324, modeling engines 326, other applications 329, etc., which may be arranged to perform actions for the embodiments described below. In one or more of the various embodiments, one or more applications may be implemented as modules or components of another application. Furthermore, in one or more of the various embodiments, applications may be implemented as operating system extensions, modules, plug-ins, etc.

[0093] Furthermore, in one or more of the various implementations, the evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329, etc., can operate in a cloud-based computing environment. In one or more of the various implementations, these applications and other applications, including the management platform, can execute within a virtual machine or virtual server (which can be managed in a cloud-based computing environment). In one or more of the various implementations, in this context, applications can flow from one physical network computer within the cloud-based environment to another physical network computer, depending on performance and scalability considerations automatically managed by the cloud computing environment. Similarly, in one or more of the various implementations, virtual machines or virtual servers dedicated to the evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329, etc., can be automatically provisioned and delegated.

[0094] Furthermore, in one or more of the various implementation schemes, the evaluation engine 322, visualization engine 324, modeling engine 326, other applications 329, etc., may reside in a virtual server running in a cloud-based computing environment, rather than being tied to one or more specific physical network computers.

[0095] In addition, the network computer 300 may also include a hardware security module (HSM) 360 for providing additional tamper-proof protection for generating, storing, or using secure / cryptographic information such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some implementations, the hardware security module may be used to support one or more standard public key infrastructures (PKIs) and may be used to generate, manage, or store key pairs, etc. In some implementations, the HSM 360 may be a standalone network computer; in other cases, the HSM 360 may be arranged as a hardware card that can be installed within the network computer.

[0096] Additionally, in one or more embodiments (not shown in the figures), the network computer 300 may include embedded logic hardware devices instead of a CPU, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable array logic (PALs), or combinations thereof. The embedded logic hardware devices can directly execute their embedded logic to perform actions. Furthermore, in one or more embodiments (not shown in the figures), the network computer may include one or more hardware microcontrollers instead of a CPU. In one or more embodiments, the one or more microcontrollers can directly execute their own embedded logic to perform actions and access their own internal memory and their own external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as system-on-a-chip (SoCs).

[0097] Descriptive logical system architecture

[0098] Figure 4 The logical architecture of a system 400 for revealing a visual illusion is illustrated according to one or more of various embodiments. In one or more of the various embodiments, system 400 may consist of various components, including one or more modeling engines, such as modeling engine 402; one or more visualization engines, such as visualization engine 404; one or more visualizations, such as visualization 406; one or more data sources, such as data source 410; one or more visualization models, such as visualization model 408; or one or more evaluation engines, such as evaluation engine 412.

[0099] In one or more of the various embodiments, the modeling engine 402 may be arranged to enable a user to design one or more visualization models that can be provided to the visualization engine 404. Therefore, in one or more of the various embodiments, the visualization engine 404 may be arranged to generate one or more visualizations based on the visualization models.

[0100] In one or more of the various implementations, the modeling engine may be configured to access one or more data sources, such as data source 410. In some implementations, the modeling engine may be configured to include a user interface that allows users to browse various data source information, data objects, etc., to design visualization models that can be used to generate visualizations of information stored in the data sources.

[0101] Therefore, in some implementations, the visualization model can be designed to provide visualizations including charts, plots, graphs, tables, patterns, styles, explanatory text, interactive elements, user interface features, etc. In some implementations, a graphical user interface can be provided to users, enabling them to interactively design the visualization model so that various elements or display objects in the visualization model can be associated with data from one or more data sources, such as data source 410.

[0102] In one or more of the various implementations, the data source, such as data source 410, may include one or more of a database, data storage, file system, etc., which may be located locally or remotely. In some implementations, the data source may be provided by another service on a network. In some implementations, one or more components (not shown) may be present that filter or otherwise provide a management view or management access to the data in the data source.

[0103] In one or more of the various implementations, the visualization model may be stored in one or more data storage devices, such as visualization model storage 408. In this example, for some implementations, visualization model storage 408 represents one or more databases, file systems, etc., used to store, protect, or index the visualization model.

[0104] In one or more of the various implementation schemes, a visualization engine, such as visualization engine 404, may be configured to parse or otherwise interpret visualization models and data from a data source to generate one or more visualizations that can be displayed to a user.

[0105] In one or more of the various implementations, an evaluation engine, such as evaluation engine 412, may be configured to evaluate or otherwise assess the visualization. Thus, in some implementations, the evaluation engine may be configured to automatically perform one or more actions to identify potential illusions or artifacts in the visualization that could mislead it. In some implementations, the evaluation engine may be configured to automatically pull additional data from the data source associated with the visualization and use a statistical model, such as an evaluation model, to assess its relevance.

[0106] Figure 5A and Figure 5B An example of how visualization can be misleading is shown.

[0107] Figure 5A A representation of a portion of visualization 500 for revealing a visualization illusion, according to one or more of various embodiments, is shown. In this example, visualization 500 can be considered a radar chart. In some cases, radar charts may be used to compare the skills of job candidates. In this example, visualization 500 shows the job skills of a hypothetical job candidate, where each axis (e.g., A, B, C, D, E, and F) represents the skill level of a particular job skill. And, in this example, the places where line 502 intersects the axes indicate the job candidate's skill level for a particular job skill. Thus, in this example, the job candidate has a higher level of ability in job skills B, C, and D, and a lower level of ability in job skills A, F, and E.

[0108] Therefore, at first glance, the Visual 500 seems to indicate that the job candidate may have a concentrated set of skills, as the job candidate appears to have skills concentrated in the upper right part of the visualization.

[0109] Figure 5BA representation of a portion of visualization 504 for revealing a visualization illusion, according to one or more of various embodiments, is shown. In this example, similar to visualization 500, visualization 504 can be considered a radar chart. Likewise, in this example, visualization 504 shows the job skills of a hypothetical job candidate, where each axis (e.g., A, B, C, D, E, and F) represents the skill level of a particular job skill. And, in this example, the places where line 506 intersects the axes indicate the job candidate's skill level for a particular job skill. Thus, similar to visualization 500, visualization 504 shows that the job candidate has a high level of ability in job skills B, C, and D, and a lower level of ability in job skills A, F, and E.

[0110] However, in this example, the positions of the C-axis and F-axis in the visualization have been swapped. Therefore, in contrast to visualization 500, visualization 504 appears to indicate that the job candidate may have a more diverse skill set (e.g., less concentrated), as the job candidate appears to have skills that seem more evenly distributed in the visualization.

[0111] In this example, visualizations 500 and 504 appear very different, even though the underlying data may be the same. Therefore, in this example, one or more visualization illusions may exist.

[0112] Figure 6A and Figure 6B Similar to Figure 5A and Figure 5B This illustrates another example of how to incorporate illusions into other reasonable display visualizations.

[0113] Figure 6A A portion of visualization 600 is shown, which includes a bar chart comparing house prices in 2001 (represented by axis 602) and prices in 2008 (year category represented by axis 604). Therefore, in this example, the visual appearance of visualization 600 appears to suggest that prices increased by 300% from 2001 to 2008, due to the relative size of the bars used in the chart.

[0114] Figure 6B A portion of visualization 610 is shown, which includes a bar chart comparing house prices in 2001 (represented by axis 612) and prices in 2008 (year categories represented by axis 614). Therefore, in this example, the visual appearance of visualization 614 appears to suggest that prices barely increased from 2001 to 2008 due to the relative size of the bars used in the chart.

[0115] In this example, Figure 6A and Figure 6BThe visual appearance seems to suggest a completely different conclusion. Figure 6A This seems to indicate a dramatic price change, while Figure 6B This seems to indicate that prices have not risen sharply. Importantly, in this example, and in... Figure 5A and Figure 5B In the examples shown, even if the underlying data or visualization type may be similar, the visualizations can have very different appearances. Visualizations that can resist this type of effect (e.g., illusions) can be considered robust visualizations.

[0116] Note that those skilled in the art will understand that visualization models or visualization engines can be arranged to generate many different types of visualizations for various purposes, depending on the design goals of the author, user, or organization. Here, visualizations 500, 504, 600, and 610 are presented as non-limiting examples to help provide clarity in the description of these innovations. Those skilled in the art will understand that these examples are sufficient to disclose at least the innovations of this document, and that visualization engines or visualization models can be arranged to generate many different visualizations for many different purposes in many fields.

[0117] Figure 7 A logical representation of a portion of an evaluation system 700 according to one or more of various embodiments is shown. In one or more of the various embodiments, system 700 may include one or more components, such as an evaluation engine 702, an evaluation model 704, a visualization model 706, a data source 708, a visualization 710, an evaluation result 712, etc. In some embodiments, the evaluation result, such as evaluation result 712, may be arranged to include additional information, such as an evaluation score 714, report information 716, etc.

[0118] In one or more of the various embodiments, the evaluation engine 702 may be arranged to evaluate the visualization 710 based on the evaluation model 704. In one or more of the various embodiments, the evaluation model may be arranged to include one or more heuristic or machine learning evaluators that can be executed to evaluate the visualization.

[0119] As discussed herein, an evaluation engine can be configured to employ one or more evaluation models and provide one or more reports with information about the degree of matching (or classification) between a given evaluation model and a visualization. Therefore, in this example, the evaluation result 712 includes scores, such as an evaluation score 714, and report information 716.

[0120] In one or more of the various implementations, the evaluation model may be arranged to provide evaluation scores representing the quality of the evaluation, such as probability scores, confidence scores, etc. In some implementations, the evaluation engine may be arranged to execute or apply the evaluation model to perform various actions to evaluate visualizations, visualization models, or data sources to determine whether a given visualization may include one or more visualization illusions.

[0121] In one or more of the various implementations, different evaluation models may employ different scoring criteria. Therefore, in some implementations, the evaluation engine may be configured to weight or standardize the evaluation scores provided by different evaluation models. In some implementations, specific standardization or weighting rules for standardizing or weighting the evaluation scores of the evaluation models may be provided via configuration information to account for local conditions.

[0122] In one or more of the various implementations, the evaluation model can be designed or customized to evaluate one or more statistical characteristics of the data associated with the potential illusion. Therefore, in one or more of the various implementations, the evaluation engine can be arranged to apply one or more evaluation models to assess whether the data associated with the visualization in the evaluation possesses one or more statistical characteristics targeted by the evaluation model. In some implementations, the evaluation model can be arranged to provide an evaluation score as a form of self-assessment, indicating how closely the data associated with the visualization matches statistical characteristics that the evaluation model can be designed to match or otherwise evaluate.

[0123] Therefore, in one or more of the various implementation schemes, the evaluation engine may be configured to automatically construct an evaluation model profile that can be used to modify or customize the evaluation report, using user activity information or user feedback. For example, if an organization's users consistently report mismatches between visualizations and evaluation results, the evaluation engine may be configured to introduce weighting rules that increase or decrease the effective evaluation score used to rank the organization's evaluation results based on user feedback information.

[0124] In one or more of the various implementations, if a user selects a visualization for illusion assessment, the assessment engine can determine one or more assessment models and apply them to the visualization being assessed, as well as the associated visualization model or data source. Therefore, non-limiting examples of assessment models will be discussed below. For the sake of brevity and clarity, this discussion is limited to a few examples; however, those skilled in the art will understand that other assessment models incorporating additional or supplementary assessment strategies may be considered.

[0125] In one or more of the various implementation schemes, within an analytical framework known as Algebraic Visualization Design (AVD), trivial changes to the data on which the visualization is based (e.g., shuffling the row order of the input data) should result in trivial changes in the resulting visualization, and significant changes in the visual appearance of the visualization should occur only as a result of corresponding significant changes in the data. Those skilled in the art will understand that AVD formalizes these explicit affirmations through commutative relations, which describe the properties of an effective visualization across underlying data transformations (not shown).

[0126] In one or more of the various implementation schemes, a definitive failure of AVD can lead to "hallucinators" (visualizations that appear distinctly different despite being supported by similar or identical data, such as...) Figure 5A , Figure 5B , Figure 6A or Figure 6B Visualizations can be categorized into "intermediate" and "confusionist" visualizations (visuals that appear identical despite being supported by distinctly different data). In some cases, visualizations may be completely unresponsive to their supporting data, serving merely as numerical decorations and creating what can be termed "non-inference" visualizations. In some cases, these AVD failures may be directly related to visualization illusions (because they result in visualizations that are fragile, unrobust, or unresponsive). However, AVD provides an analytical language and visualization specification for data manipulation, enabling evaluation models to be evaluated automatically. Therefore, in some implementations, AVD can provide a useful framework for designing tests that can detect failures requiring minimal domain knowledge. For example, in some cases, an AVD-based evaluation model might simply induce trivial or non-trivial data variations and examine the corresponding variations in the visualization of the results.

[0127] In one or more of the various implementations, the evaluation engine may be configured to employ deformation testing as a mechanism for validating or otherwise evaluating individual visualizations. In some implementations, the evaluation engine may be configured to employ an evaluation model capable of performing deformation operations: modifications to the data and modifications to the design specifications. Thus, in some implementations, the evaluation engine can be enabled to evaluate multiple types of visualizations without requiring extensive knowledge of the presented visualizations.

[0128] In one or more of the various implementations, changing the order of the input data should not alter the presented visualization. Therefore, in some implementations, the evaluation model may be arranged to employ a detection technique such as a pixel differencing algorithm that limits the maximum number of pixels that differ between the presented images based on the order of the input rows. Thus, in some implementations, an evaluation model that provides this type of testing can allow for the detection of overdraw. Note that in some implementations, while not all drawing must indicate a visual illusion, it can be useful to alert the user to its presence in many visualization types.

[0129] In some implementations, the obvious patterns in the visualization should be robust: that is, specific relationships should persist with minor changes. Therefore, one or more evaluation models can be arranged to focus on providing a phantom detector for the bar chart. In some implementations, such evaluation models can test variability in a relatively parameter-free manner across a wide range of data distributions and complexities. In some implementations, the evaluation model can be arranged to identify which input rows need to be modified by creating a backward-origin concept that links each aggregation tag to the input tuples describing it. Thus, the evaluation model may be able to identify when the visualization might rely on outliers or a small number of different records causing differences between aggregations.

[0130] In one or more of the various implementations, the evaluation model may be arranged to evaluate aggregations that can be commonly and advantageously used to summarize information. However, in some cases, aggregations may mask data problems such as varying record numbers, sampling problems, duplicate records, etc. Therefore, in some implementations, the evaluation model may be arranged to evaluate the robustness of the metric in the context of potentially obsolete data. In some implementations, the evaluation model may be configured to define a minimum number of records that make up an aggregation and to shrink the number of records constituting all other tags to that minimum number through non-replacement sampling. In some implementations, the evaluation model may employ a randomization process to probe aggregations in the visualization. Thus, if all aggregations have similar sample sizes, and those sample sizes are large enough, and the aggregation method is robust enough to extreme values, the process should produce reasonably similar visualizations. Therefore, in some implementations, the evaluation model may be arranged to detect variability caused by sampling problems and other problems related to varying numbers of records.

[0131] Figure 8 A logical representation of a portion of a user interface 800 for revealing a visual illusion, according to one or more of various embodiments, is shown. In some embodiments, the user interface 800 may be arranged to include one or more panels, such as panel 802, panel 804, panel 806, panel 808, description 810, etc.

[0132] In this example, for one or more of the various implementations, panel 802 may be arranged to display a ranked list of phantom assessment information evaluated by the assessment engine for visualization. In some implementations, each report item may include information describing the type of phantom that may be suspected, including assessment scores, if available.

[0133] Similarly, in some implementations, if the evaluation engine is unable to identify a potential illusion, panel 802 may be arranged to display a description of the effect.

[0134] In one or more of the various embodiments, panel 804 may be arranged to include additional explanations or visualizations that can be associated with the visualizations evaluated by the evaluation engine. In some embodiments, a panel such as panel 806 may include additional explanatory information related to the detected hallucinations, or one or more visualizations that can help explain the hallucinations to the user. Furthermore, in some embodiments, a panel such as panel 808 may include visualizations in the evaluation or other visualizations that can help the user “see” the effect of the identified hallucinations. For example, panel 808 may include one or more optional visualizations that help illustrate the effect of the detected hallucinations.

[0135] In one or more of the various implementation schemes, the evaluation engine, visualization engine, etc., may be configured to use specific visualization models or explanatory text / narrative templates for different evaluation models. In some implementation schemes, the visualization model or explanatory / narrative template for the evaluation model may be defined in configuration information. Therefore, the report information (explanatory / narrative text) or visualization included in the user interface 800 may be customized for specific organizations, regions, etc.

[0136] In one or more of the various implementations, the evaluation engine or visualization engine may be configured to determine part or all of the content or style of the user interface 800 via configuration information. Therefore, in some implementations, panel layouts, explanatory text, templates, visualizations, etc., can be configured differently depending on the local environment.

[0137] Common operations

[0138] Figures 9-12 This refers to a general operation for revealing a visual illusion, according to one or more of various implementations. In one or more of the various implementations, combined with... Figures 9-12 The described processes 900, 1000, 1100, and 1200 can be performed by a single network computer (or network monitoring computer) (e.g.) Figure 3 The processes or portions thereof are implemented or executed by one or more processors on a network computer (300). In other embodiments, these processes or portions thereof may be implemented or executed by multiple network computers (e.g., Figure 3The network computer 300) implements or executes, or is implemented or executed thereon. In other embodiments, these processes, or portions thereof, may be implemented or executed by, or on, one or more virtualized computers, such as virtualized computers in a cloud-based environment. However, the embodiments are not limited thereto, and various combinations of network computers, client computers, etc., may be used. Furthermore, in one or more of the various embodiments, the combination of Figures 9-12 The described process can be used based on, for example, combination Figures 4-8 At least one of the described multiple implementations or architectures is used to reveal the visualization illusion. Furthermore, in one or more of the various implementations, some or all of the actions performed by processes 900, 1000, 1100, and 1200 may be partially performed by the evaluation engine 322, visualization engine 324, modeling engine 326, etc.

[0139] Figure 9 A flowchart outlining a process 900 for revealing a visualization illusion according to one or more of various embodiments is shown. Following the start box, at box 902, in one or more of the various embodiments, a visualization engine may be arranged to generate one or more visualizations based on one or more visualization models or data sources. As described above, a visualization system may be arranged to include one or more modeling engines, one or more data sources, one or more visualization engines, etc., which may be arranged to generate visualizations based on one or more visualization models and data provided by one or more data sources.

[0140] In box 904, in one or more of the various implementations, optionally, the evaluation engine may be arranged to determine one or more visualizations that may be of interest.

[0141] In one or more of the various implementation schemes, the user or visualization author may be able to select one or more visualizations for illusion analysis. For example, a user interface may be provided that allows the author to browse one or more previously generated visualizations.

[0142] Similarly, in one or more of the various implementations, the evaluation engine may be configured to automatically evaluate visualizations when they are created. In some implementations, one or more visualizations may be identified for evaluation based on their membership in one or more visualization categories or types.

[0143] In one or more of the various implementations, the evaluation engine may be configured to apply various heuristics or filters to determine which visualizations used for illusion should be evaluated. Thus, in some implementations, the evaluation engine may be configured to use one or more rules or instructions provided via configuration information to determine whether visualizations used for illusion should be automatically evaluated.

[0144] In some implementations, the evaluation engine may be configured to evaluate one or more visualizations based on criteria selected or recommended via configuration information.

[0145] In box 906, in one or more of the various implementations, the evaluation engine may be arranged to analyze one or more visualizations of interest based on one or more evaluation models. In one or more of the various implementations, the evaluation engine may be arranged to execute one or more evaluation models to evaluate one or more visualizations to determine whether there may be visualization illusions hidden in one or more visualizations.

[0146] At box 908, in one or more of the various embodiments, the evaluation engine may be arranged to generate one or more illusion evaluation reports, which may include one or more visualizations demonstrating the discovered illusions (if any). As described above, in some embodiments, the evaluation report may include an interactive user interface that allows a user to review potential visual illusions that may have been discovered by the evaluation engine.

[0147] Next, in one or more of the various implementation schemes, control may be returned to the calling procedure.

[0148] Figure 10 A flowchart of a process 1000 for evaluating visualizations to discover visualization illusions, according to one or more of various embodiments, is shown. Following the start box, in box 1002, in one or more of the various embodiments, one or more evaluation models and one or more visualizations may be provided to the evaluation engine. As described above, in some embodiments, one or more evaluation models may be defined to discover visualization illusions. Therefore, in some embodiments, the evaluation engine may be arranged to obtain one or more evaluation models from a data storage device, etc. In some embodiments, the evaluation engine may be arranged to determine a particular evaluation model based on rules, conditions, etc., which can be provided via configuration information.

[0149] In box 1004, in one or more of the various implementations, the evaluation engine may be configured to evaluate one or more evaluation models based on tags, visualization models, data sources, etc. In one or more of the various implementations, the evaluation model may include, or be associated with, rules, conditions, computer-readable instructions, etc., which the evaluation engine may execute or apply to evaluate whether the visualization may include one or more visualization illusions.

[0150] In decision box 1006, in one or more of the various embodiments, if there may be more evaluation models to evaluate, control may loop back to box 1004; otherwise, control may proceed to box 1008. In some embodiments, the evaluation engine may be configured to terminate the evaluation early by omitting or skipping one or more evaluation models. For example, in some embodiments, the evaluation engine may be configured to stop evaluating additional evaluation models if it can be considered that the previously evaluated evaluation models provided sufficient phantom analysis. In some embodiments, one or more evaluation models may become redundant or meaningless depending on whether one or more other evaluation models identify or exclude some phantoms.

[0151] In one or more of the various implementations, the evaluation engine may be configured to correlate evaluation scores with evaluation results provided by the evaluation model. In some implementations, the evaluation model may be configured to provide evaluation scores that indicate how well the evaluation model matches a particular visual illusion.

[0152] Furthermore, in one or more of the various embodiments, the evaluation engine may be arranged to maintain another set of evaluation scores, which may be combined with the evaluation scores provided by each evaluation model used to identify the visual illusion. Thus, in some embodiments, the overall evaluation score associated with the discovered illusion may be a combination of scores provided by the evaluation models and scores provided by the evaluation engine.

[0153] For example, in some implementations, the evaluation score provided by the evaluation engine may be generated based on other factors or metrics that are unavailable or irrelevant to the evaluation model. In some implementations, such factors or metrics may be based on user feedback, organizational needs (e.g., configuration information), metrics associated with organizational or community activities, etc.

[0154] In box 1008, in one or more of the various embodiments, the evaluation engine may be arranged to filter or sort the results provided by the evaluation model. In one or more of the various embodiments, the evaluation model may be arranged to provide one or more evaluation scores that represent a self-assessment of the quality of the illusion determination.

[0155] Furthermore, in one or more of the various implementation schemes, the evaluation engine can be configured to modify the evaluation scores provided by the evaluation model to provide a final evaluation score.

[0156] Therefore, in one or more of the various implementations, the evaluation engine may be configured to filter one or more evaluation models based on rules, conditions, etc., to eliminate one or more identified phantoms. For example, in some implementations, the evaluation engine may be configured to automatically exclude one or more phantoms associated with evaluation scores below a defined threshold.

[0157] Furthermore, in one or more of the various implementations, other metrics or characteristics of one or more of the visualization model, data source, data, users, organization, etc., can be incorporated into conditions, rules, etc., which can be executed to filter out one or more evaluations or illusions. For example, in some implementations, the filter may include conditions associated with one or more user roles. Thus, for example, the filter can be configured to be more inclusive of some user roles or less inclusive of others. Therefore, in this example, the number of potential illusions included in the evaluation report could be increased for data scientists and decreased for ordinary users.

[0158] In some implementations, other metrics or features used for filtering can be based on a variety of sources, including organizational preferences, user preferences, location, user input, and other configuration information.

[0159] Similarly, in one or more of the various implementations, the evaluation engine may be configured to rank one or more discovered illusions based on various conditions, rules, metrics, or features. For example, in some implementations, discovered illusions (e.g., latent illusions) may be ranked based on evaluation scores, etc. Furthermore, for example, in some implementations, one or more descriptions may be ranked based on user preferences, organizational preferences, etc.

[0160] Similar to filters, in some implementations, the various conditions, rules, metrics, or features used for interpreting ranking can be based on a variety of sources, including organizational preferences, user preferences, localization, user input, other configuration information, etc.

[0161] In box 1010, in one or more of the various embodiments, the evaluation engine may be configured to generate a hallucination evaluation report. In one or more of the various embodiments, the hallucination evaluation report may include various information, including a ranking list of potential or discovered hallucinations, a description of one or more characteristics of various hallucinations, visualizations showing how the hallucinations appear in different contexts or views, etc.

[0162] Next, in one or more of the various implementation schemes, control is returned to the calling program.

[0163] Figure 11 A flowchart of process 1100 for determining a visualization that may include one or more visualization illusions, according to one or more of various embodiments, is shown. Following the start box, in box 1102, in one or more of the various embodiments, the raw visualization can be provided to the evaluation engine. Therefore, in one or more of the various embodiments, a visualization generated based on a visualization model, one or more data models, one or more data sources, etc., can be provided to the evaluation engine.

[0164] In one or more of the various implementation schemes, the evaluation engine may be provided with models, data, or information that can be used to generate visualizations.

[0165] In box 1104, in one or more of the various embodiments, the evaluation engine may be arranged to determine the next evaluation model. As described above, in some embodiments, there may be several different evaluation models available for discovering illusions in a given visualization. In some embodiments, the evaluation models may be provided in any order. However, in some embodiments, different types or categories of evaluation models may be grouped so that they can be used together or nearly simultaneously. For example, in some embodiments, there may be one or more classes of evaluation models that require one or more of the same processing actions as part of determining visualization illusions. Therefore, in some embodiments, evaluation models that may depend on some of the same data processing may be executed together or nearly simultaneously.

[0166] Similarly, in some implementations, more than one evaluation model can point to the same or similar type of phantom. Therefore, in some implementations, if the evaluation models target the same type of phantom, models that consume fewer resources (or are faster) can be prioritized for application before those that consume more resources are executed.

[0167] Furthermore, in some implementations, the evaluation engine may be configured to rank evaluation models based on one or more recentity rules. In some implementations, the evaluation engine may be configured to track or record the phantom history of individual authors, data sources, data models, organizations, visualization classes, etc. Therefore, in one or more of the various implementations, phantom history may be taken into account to rank or classify evaluation models. For example, in some implementations, if a particular visualization author has a history of introducing a particular class of phantoms, an evaluation model for discovering such phantoms may be executed first.

[0168] However, in some implementations, the evaluation engine can be configured to use rules, instructions, etc., provided through configuration information (which can be used to describe the local environment or conditions).

[0169] In box 1106, in one or more of various embodiments, the evaluation engine may be configured to render one or more test visualizations based on the current evaluation model. As described above, the evaluation model may include one or more instructions for generating test visualizations based on the same data used to generate the original visualization. For example, in some embodiments, the evaluation model may define rules to generate a modified visualization model that can be used to generate test visualizations, such as swapping axes, adjusting baseline values, aggregating sampled data values ​​used, changing the plotting order of different markers, etc., or combinations thereof. In some embodiments, additional fields from the data source may be included or replaced to generate test visualizations.

[0170] In box 1108, in one or more of the various implementations, the evaluation engine may be arranged to compare the original visualization with the test visualization to determine whether the original visualization may contain phantoms.

[0171] In one or more of the various implementation schemes, the specific comparisons and associated criteria may depend on a given evaluation model. Therefore, in some implementation schemes, as described above, the evaluation model may include or reference one or more rules, instructions, conditions, parameters, etc., which determine how to perform the comparisons and how to interpret the results.

[0172] In one or more of the various implementations, the comparison may include generating or presenting a flattened image (e.g., a bitmap, etc.), which may enable various machine vision or statistical analyses to compare the original visualization with the test visualization.

[0173] In some implementations, the evaluation engine may be configured to perform conventional machine vision methods to quantify various differences between the original visualization and the test visualization. For example, in some implementations, if the visualization is robust (e.g., a visualization without phantoms), small changes to the data or visualization model should be expected to have a small change to the appearance of the visualization. Similarly, in many cases, non-substantial changes to the appearance of other robust visualizations (e.g., colors, plotting order, orientation, etc.) should have a small change to the appearance of the visualization.

[0174] At decision box 1110, in one or more of the various embodiments, if the deviation between the original visualization and one or more test visualizations exceeds a defined threshold, control can proceed to box 1112; otherwise, control can proceed to decision box 1114. As described above, the evaluation model can be arranged to generate an evaluation score that may represent the probability or confidence score of a particular phantom being detected.

[0175] In box 1112, in one or more of the various embodiments, the evaluation engine may be arranged to associate evaluation information, including evaluation scores, with the original visualization. Thus, in some embodiments, the associated evaluation information may be provided to the user or author of the evaluated visualization. For example, if a phantom can be detected, a report may be generated or provided to one or more users indicating the type of phantom, data, or changes in appearance that enable the detection of the phantom.

[0176] In one or more of the various implementations, the evaluation model may be arranged to provide evaluation scores, etc., which may indicate a determined strength or quality (e.g., confidence score). Therefore, in one or more of the various implementations, the evaluation engine may use evaluation scores provided by the evaluation model, rather than a specific deviation value between the original visualization and the test visualization.

[0177] In decision block 1114, in one or more of the various implementations, if there may be more evaluation models to be performed, control may loop back to block 1104; otherwise, control may return to the calling procedure.

[0178] Figure 12 A flowchart of process 1200 for evaluating visualizations that may include one or more visual illusions, according to one or more of various embodiments, is shown. Following the start box, in box 1202, in one or more of the various embodiments, one or more evaluation models and one or more visualizations may be provided to the evaluation engine. As described above, in some embodiments, one or more evaluation models may be defined for detecting visual illusions. Therefore, in some embodiments, the evaluation engine may be arranged to obtain one or more evaluation models from a data storage device, etc. In some embodiments, the evaluation engine may be arranged to determine a particular evaluation model based on rules, conditions, etc., which may be provided via configuration information.

[0179] In one or more of the various implementations, the evaluation engine can be arranged in a pipeline employing different evaluation models to evaluate whether a visualization can be associated with one or more illusions. In some implementations, each stage of the pipeline can be associated with a different stage of information processing that leads to the final visualization. In some implementations, each stage of the pipeline may allow the introduction of visualization illusions. Therefore, in some implementations, evaluation models can be designed or configured for each stage of the pipeline.

[0180] In box 1204, in one or more of the various embodiments, the evaluation engine may be arranged to execute one or more evaluation models that can be directed to data management. Thus, in one or more of the various embodiments, the evaluation model may be arranged to evaluate various characteristics of the management of the data used to generate visualizations. In some embodiments, this may include evaluating various characteristics of the source data on which the visualization is based, such as missing or duplicate records, outliers, spelling errors, drill-down bias, etc.

[0181] In one or more implementations, missing or duplicate records may be a concern because users may often assume that there is one and only one entry for each piece of data associated with the visualization. However, in some cases, errors in data entry or integration can lead to missing or duplicate values, resulting in inaccurate aggregation or grouping.

[0182] In one or more of the various implementations, outliers may be of concern because many forms of analysis can assume that the data are of similar size and generated by similar processes. Therefore, in some implementations, outliers, whether in the form of erroneous or unexpected extreme values, can significantly affect aggregations and render the assumptions behind many statistical tests and summaries unreliable.

[0183] In one or more of the various implementations, spelling errors may be a concern because, in some cases, string columns can be interpreted as categorical data for aggregation purposes. Therefore, in some implementations, if interpreted in this way, typos or inconsistent spelling and capitalization can create false categories, remove important data from aggregation queries, etc.

[0184] In one or more of the various implementations, drill-down bias may be a concern because users assume the order of the survey data should not affect their conclusions. However, in some cases, by filtering out less interpretable or more relevant variables first, the full range of the influence of later variables may be obscured. Therefore, in some implementations, this may result in insights that are only relevant to a small portion of the data, while those insights may be applicable to a larger whole.

[0185] In box 1206, in one or more of the various implementations, the evaluation engine may be arranged to perform one or more evaluation models that may be directed to data processing or data transformation.

[0186] Therefore, in one or more of the various implementations, evaluation models can be deployed to analyze whether the data processing involves varying numbers of records in grouping partitions. In some implementations, these may be of concern because certain summary statistics, including aggregation, can be sensitive to sample size. However, the number of records aggregated into a single visual label can vary significantly. In some cases, this mismatch can mask sensitivity and interfere with per-label comparisons. Furthermore, in some implementations, if combined with aggregation at different levels, it can lead to counterintuitive results, such as Simpson's Paradox.

[0187] In one or more of the various implementations, the evaluation model may be arranged to determine the scope of filtered data that can be used in visualization. In some implementations, filtering and subsettings are meant as tools for removing irrelevant data or allowing users to focus on specific areas of interest. However, if such filtering is too aggressive, or if users focus on individual examples rather than overall trends, such filtering may promote erroneous conclusions or biased perceptions of relationships between variables. Therefore, in some implementations, failure to maintain a broader dataset within context can lead to the Texas Sharpshooter Fallacy or other forms of hacking.

[0188] In one or more of the various implementations, the evaluation model may be arranged to determine how degrees of freedom might affect the extent of visualization. In some implementations, this may be a concern because the authors of the visualizations may have considerable flexibility in how the users analyze the data. In some implementations, such “researcher degrees of freedom” may produce conclusions that are highly specific to the choices made by the analyst, or maliciously promote “p-hacking,” in which the analyst searches the parameter space to find the best support for a predetermined conclusion. In some implementations, a related problem that may be encountered is the “multiple comparison problem,” in which users make so many choices that at least one configuration may appear important only by chance, even if there is no strong signal in the data.

[0189] In one or more of the various implementations, the evaluation model may be arranged to determine the extent to which misattribution may affect the visualization. In some implementations, this may be a concern because there are many strategies for handling missing or incomplete data, including the estimation of new values. Therefore, in some implementations, how values ​​are estimated, and then how those estimates are visualized in the context of the remaining data, can influence how the data is perceived, potentially producing spurious trends or grouping discrepancies, which may simply be artifacts of how missing values ​​were handled prior to visualization.

[0190] In box 1208, in one or more of the various embodiments, the evaluation engine may be arranged to evaluate the visualization. In one or more of the various embodiments, the evaluation model may be arranged to determine whether the appearance of the visualization contributes to one or more visualization illusions.

[0191] In one or more of the various implementations, the evaluation model may be arranged to evaluate whether the visualization includes discontinuous visualizations. In some implementations, these may be of concern because users expect the graph to appear as a mapping between data and images. In some cases, visualizations used as decoration (where the labels are unrelated to the data) present non-information that may be misinterpreted as true information. Therefore, in some cases, even if the data is accurate, additional unreasonable annotations can create a misleading impression, such as decorating irrelevant data with a false best-fit line, etc.

[0192] In one or more of the various implementations, the evaluation model may be arranged to assess whether the visualization includes overdrawing. In some implementations, this may be a concern because users typically expect to be able to clearly identify individual markers in a visualization and expect a visual marker to correspond to a single value or aggregate value. However, in some cases, overlapping markers can hide the internal structure of a distribution or mask potential data quality issues.

[0193] In one or more of the various implementations, the evaluation model can be arranged to assess whether the visualization includes hidden uncertainties. In some implementations, this may be a concern because visualizations that fail to indicate the presence of uncertainty risks may give a false impression and could lead to extreme distrust of the data if the reader realizes that the information is not clearly presented. In some cases, there is also a tendency to mistakenly assume that the data is of high quality or complete, even without evidence of such accuracy.

[0194] In one or more of the various implementations, the evaluation model may be arranged to assess whether the visualization includes manipulation of scale. In some implementations, this may be a concern because users assume that the axes and scales of the visualization directly represent quantitative information. However, such manipulation of scale (e.g., by flipping them from their typically assumed orientation, truncating or expanding them relative to the data range, using nonlinear transformations, using dual axes, etc.) may cause viewers to misinterpret the data in the chart, for example by exaggerating correlations, exaggerating effect sizes, misinterpreting effect directions, etc.

[0195] In box 1210, in one or more of the various implementations, the evaluation engine may be arranged to evaluate reader effects that can be associated with visualization.

[0196] In one or more of the various implementations, the evaluation model may be arranged to assess whether the visualization includes a basic rate of deviation. In some implementations, this may be of concern because users may assume that unexpected values ​​in the visualization symbolize reliable differences. However, in some cases, the reader may not be aware of the relevant basic rate: either the relative probability of being considered an unexpected value or the false discovery rate of the entire analysis process.

[0197] In one or more of the various implementations, the evaluation model may be arranged to assess whether a visualization depends on one or more inaccessible visualizations. In some implementations, this may be of concern because the visualization author may assume that the user is a homogeneous group. However, some people may read visualizations differently and rely on underlying perceptual abilities and cognitive context, which may be overlooked by the visualization author. Therefore, in some implementations, insufficient attention to these differences may lead to erroneous communication. For example, in some implementations, a user with color vision deficiencies may interpret them as identical when the visualization author intends to separate them.

[0198] In one or more of the various implementations, the evaluation model may be arranged to assess whether the visualization may include an anchoring effect. In some implementations, this may be a concern because the initial frame of information tends to guide subsequent judgments. In some cases, this may lead readers to give unnecessarily rhetorical weight to early observations, causing them to underestimate or distrust later observations.

[0199] In one or more of the various implementations, the evaluation model may be arranged to assess whether visualization promotes bias in interpretation. In some implementations, this may be a concern because each viewer encounters visualization with their own preconceived notions, biases, and cognitive frameworks. Therefore, in some implementations, without careful consideration of these biases, various cognitive biases, such as counterproductive effects or confirmation biases, can lead viewers to anchor only to data (or data readings) that support their preconceived notions, reject data that does not align with their views, and often ignore the more holistic picture of the strength of evidence.

[0200] In box 1212, in one or more of the various implementations, the evaluation engine may be arranged to execute one or more other evaluation models that may point to the same or other visualization phantom sources.

[0201] Therefore, in some implementations, the evaluation engine may be configured to employ one or more rules, instructions, conditions, etc., which may be provided by configuration information to describe local conditions, including acquiring or otherwise determining one or more custom or localized evaluation models.

[0202] Next, in one or more of the various implementation schemes, control returns to the calling procedure.

[0203] It will be understood that each box in each flowchart illustration, and combinations of boxes in each flowchart illustration, can be implemented by computer program instructions. These program instructions can be provided to a processor to generate a machine, such that instructions executing on the processor create means for implementing the actions specified in each flowchart box or blocks. The computer program instructions can be executed by a processor to cause the processor to perform a series of operational steps to produce a computer-implemented process, such that instructions executing on the processor provide steps for implementing the actions specified in each flowchart box or blocks. The computer program instructions can also cause at least some of the operational steps shown in the boxes of each flowchart to be executed in parallel. Furthermore, some of the steps can also be executed across more than one processor, as may occur in a multiprocessor computer system. Moreover, without departing from the scope or spirit of the invention, one or more boxes or combinations of boxes in each flowchart illustration can also be executed simultaneously with other boxes or combinations of boxes, or even in a different order than shown in the illustration.

[0204] Therefore, each block in each flowchart illustration supports a combination of means for performing a specified action, a combination of steps for performing a specified action, and a program instruction means for performing a specified action. It will also be understood that each block in each flowchart illustration and the combination of blocks in each flowchart illustration can be implemented by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs the specified action or steps. The foregoing examples should not be construed as limiting or exhaustive, but rather as illustrative use cases to illustrate implementations of at least one of various embodiments of the invention.

[0205] Furthermore, in one or more embodiments (not shown in the figures), embedded logic hardware devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable array logic (PALs), or combinations thereof, can be used instead of a CPU to execute the logic in the illustrative flowchart. The embedded logic hardware device can directly execute its embedded logic to perform actions. In one or more embodiments, a microcontroller can be arranged to directly execute its own embedded logic to perform actions and access its own internal memory and its own external input and output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as a system-on-a-chip (SoC).

Claims

1. A method for managing data visualization using one or more processors, said one or more processors executing instructions to perform actions, the method comprising: Provide one or more visualizations based on data from the data source; Identify one or more visualizations of potential interest from the one or more visualizations, which may include one or more illusions, and wherein the one or more illusions are visually misleading representations of data in the one or more visualizations; One or more evaluation models are used to determine one or more evaluation results, said one or more evaluation results corresponding to the probability that a corresponding visualization among the one or more visualizations of potential interest meets the evaluation criteria, said evaluation criteria including criteria for detecting one or more phantoms in the corresponding visualization, including: Based on the corresponding dataset from the data source corresponding to the corresponding visualization, generate one or more test visualizations; Compare the differences between the one or more test visualizations and the corresponding visualizations; and Based on the comparison, one or more confidence scores are determined for detecting one or more phantoms in the corresponding visualization, wherein the value of the confidence score among the one or more confidence scores is proportional to the magnitude of the difference in the comparison; Based on the one or more confidence scores, the one or more evaluation results are ranked; and Provide a report that includes a ranking list of the results of the one or more assessments.

2. The method of claim 1, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: Evaluate the data associated with the corresponding visualization to identify one or more of the following: missing or duplicate records, spelling errors, drill-down bias, varying numbers of grouped records, and misleading data selection.

3. The method of claim 1, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: The corresponding visualization is evaluated to identify one or more types of illusion, including one or more of discontinuous visualization, overdrawing, hidden uncertainty, and scale manipulation.

4. The method of claim 1, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: Evaluate the data associated with the corresponding visualization, the data being included in the data source and omitted from the corresponding visualization; and Based on the assessment, a portion of the one or more assessment results is generated.

5. The method of claim 1, wherein providing the report further comprises: Generate one or more additional visualizations associated with one or more of the one or more evaluation models, the one or more evaluation results, and the one or more illusions.

6. A computer-readable storage medium comprising instructions for managing visualization, wherein the instructions, when executed by one or more processors, perform actions including: Provide one or more visualizations based on data from the data source; Identify one or more visualizations of potential interest from the one or more visualizations, which may include one or more illusions, and wherein the one or more illusions are visually misleading representations of data in the one or more visualizations; One or more evaluation models are used to determine one or more evaluation results, said one or more evaluation results corresponding to the probability that a corresponding visualization among the one or more visualizations of potential interest meets the evaluation criteria, said evaluation criteria including criteria for detecting one or more phantoms in the corresponding visualization, including: Based on the corresponding dataset from the data source corresponding to the corresponding visualization, generate one or more test visualizations; Compare the differences between the one or more test visualizations and the corresponding visualizations; and Based on the comparison, one or more confidence scores are determined for detecting one or more phantoms in the corresponding visualization, wherein the value of the confidence score among the one or more confidence scores is proportional to the magnitude of the difference in the comparison; Based on the one or more confidence scores, the one or more evaluation results are ranked; and Provide a report that includes a ranking list of the results of the one or more assessments.

7. The medium of claim 6, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: Evaluate the data associated with the corresponding visualization to identify one or more of the following: missing or duplicate records, spelling errors, drill-down bias, varying numbers of grouped records, and misleading data selection.

8. The medium of claim 6, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: The corresponding visualization is evaluated to identify one or more types of illusion, including one or more of discontinuous visualization, overdrawing, hidden uncertainty, and scale manipulation.

9. The medium of claim 6, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: Evaluate the data associated with the corresponding visualization, which is included in the data source and omitted from the one or more visualizations; and Based on the assessment, a portion of the one or more assessment results is generated.

10. The medium of claim 6, wherein providing the report further comprises: Generate one or more additional visualizations associated with one or more of the one or more evaluation models, the one or more evaluation results, and the one or more illusions.

11. A computer system, comprising: A memory that stores at least instructions; and One or more processors execute the instructions, the instructions performing actions including: Provide one or more visualizations based on data from the data source; Identify one or more visualizations of potential interest from the one or more visualizations, which may include one or more illusions, and wherein the one or more illusions are visually misleading representations of data in the one or more visualizations; One or more evaluation models are used to determine one or more evaluation results, said one or more evaluation results corresponding to the probability that a corresponding visualization among the one or more visualizations of potential interest meets the evaluation criteria, said evaluation criteria including criteria for detecting one or more phantoms in the corresponding visualization, including: Based on the corresponding dataset from the data source corresponding to the corresponding visualization, generate one or more test visualizations; Compare the differences between the one or more test visualizations and the corresponding visualizations; and Based on the comparison, one or more confidence scores are determined for detecting one or more phantoms in the corresponding visualization, wherein the value of the confidence score among the one or more confidence scores is proportional to the magnitude of the difference in the comparison; Based on the one or more confidence scores, the one or more evaluation results are ranked; and Provide a report that includes a ranking list of the results of the one or more assessments.

12. The computer system of claim 11, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: Evaluate the data associated with the corresponding visualization to identify one or more of the following: missing or duplicate records, spelling errors, drill-down bias, varying numbers of grouped records, and misleading data selection.

13. The computer system of claim 11, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: The corresponding visualization is evaluated to identify one or more types of illusion, including one or more of discontinuous visualization, overdrawing, hidden uncertainty, and scale manipulation.

14. The computer system of claim 11, wherein determining the one or more evaluation results using the one or more evaluation models further comprises: Evaluate the data associated with the corresponding visualization, which is included in the data source and omitted from the one or more visualizations; and Based on the assessment, a portion of the one or more assessment results is generated.

15. The computer system of claim 11, wherein providing the report further comprises: Generate one or more additional visualizations associated with one or more of the one or more evaluation models, the one or more evaluation results, and the one or more illusions.

Images