Variable-length character encoding using concurrent processing

Abstract

Embodiments are directed to managing character encoding. A plurality of characters, each encoded as a code unit based on a character code, may be provided such that the code unit of each character represents a code point of a character encoding scheme. An encoding model may be determined based on the character code, one or more processor functions, and a target character code. A process function may be used to convert the code units to target code units based on the encoding model, such that the target code units are based on the target character code, and such that the target code units encode the code points of each character. A plurality of target characters may be provided to a target stream such that each target character may be encoded as a target code unit.

Description

[Technical Field]

[0001] The present invention relates generally to character encoding, and more particularly, but not exclusively, to using concurrent processing to improve encoding / decoding performance. [Background technology]

[0002] Organizations generate and collect ever-increasing amounts of data. This data may be associated with different parts of the organization, such as consumer activities, manufacturing activities, customer service, server logs, etc. For various reasons, it may be inconvenient for such organizations to effectively utilize vast collections of data. In some cases, the volume of data may make it difficult to effectively utilize the collected data to improve business practices. In some cases, organizations may generate visualizations of some or all of their data. Using visualizations to represent this data may enable organizations to improve their understanding of critical business operations and help them monitor key performance indicators. In some cases, visualizations, configuration information, or other information may include character data, such as annotations, labeling, narratives, field names, etc., that may be stored using various character encoding schemes. In some cases, it may be advantageous to convert characters encoded in one character code to another character code. Therefore, it may be advantageous to reduce resource costs that may be associated with converting characters on the fly or otherwise. It is with respect to these and other considerations, therefore, that the present invention has been made. [Brief explanation of the drawings]

[0003] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following drawings, in which like reference numerals refer to like parts throughout the various views unless otherwise specified. For a better understanding of the present invention, reference is made to the following detailed description, which should be read in conjunction with the accompanying drawings. [Figure 1]FIG. 1 illustrates a system environment in which various embodiments may be implemented. [Figure 2] 1 illustrates a schematic embodiment of a client computer. [Figure 3] 1 illustrates a schematic embodiment of a network computer. [Figure 4] 1 illustrates a logical architecture of a system for encoding variable length characters using concurrent processing according to one or more of various embodiments. [Figure 5] 1 illustrates a logical schematic diagram of a portion of a system for encoding variable length characters using concurrent processing, according to one or more of various embodiments. [Figure 6] 1 illustrates a logical schematic diagram of a portion of an encoding engine for encoding variable length characters using concurrent processing, according to one or more of various embodiments. [Figure 7] 1 illustrates a logical schematic diagram of a table comparing character codes for encoding variable length characters using concurrent processing, according to one or more of various embodiments. [Figure 8] 1 depicts a high-level flowchart of a process for encoding variable-length characters using concurrent processing, according to one or more of various embodiments. [Figure 9] 1 illustrates a flowchart of a process for converting UTF-16 encoded ASCII characters to UTF-8 in accordance with one or more of various embodiments. [Figure 10] 1 illustrates a flowchart of a process for converting UTF-8 encoded CJK characters to UTF-16 encoded CJK characters in accordance with one or more of various embodiments. [Figure 11] 1 provides an overview of some character encoding standards that may be followed in one or more of various embodiments. [Figure 12] 1 illustrates a flowchart for a sequential single code unit UTF-8 source to target (optional) process according to one or more of various embodiments. [Figure 13]1 illustrates a flowchart for a process for signature 32-bit UTF-8 source to target (optional) according to one or more of various embodiments. [Figure 14] 1 illustrates a flowchart for an SSE4 128-bit UTF_8 source to target (optional) process according to one or more of various embodiments. [Figure 15] 1 illustrates a flowchart for an AVX-512 512-bit UTF-8 source to (any) target process according to one or more of various embodiments. [Figure 16] Here is a flowchart of the process from a sequential single code unit (currently) UTF-16 source to a UTF-8 target: [Figure 17] Here is a flowchart of the process from an SSE4 128-bit UTF-16 source to a UTF-8 target: [Figure 18] 1 illustrates a flowchart of a process for AVX-512 512-bit UTF-16 source to UTF-8 target in accordance with one or more of various embodiments. [Figure 19] This shows a flowchart of the process for sequential single code units (currently) from a UTF-32 source to a UTF-8 target. [Figure 20] 1 illustrates a flowchart of a process for an SSE4 128-bit UTF-32 source to a UTF-8 target in accordance with one or more of various embodiments. [Figure 21] 1 illustrates a flowchart for an AVX 512-bit UTF-32 source to UTF-8 target process in accordance with one or more of various embodiments. [Figure 22] This shows the flowchart for Process 0 for sequential single code units (currently) from a UTF-16 source to a UTF-32 target. [Figure 23] Here is a flowchart of the process from an SSE2 128-bit UTF-16 source to a UTF-32 target: [Figure 24]1 illustrates a flowchart of a process for AVX-512 512-bit UTF-16 source to UTF-32 target in accordance with one or more of various embodiments. [Figure 25] 1 shows a flowchart of the sequential single code unit UTF-32 source to UTF-16 target process. [Figure 26] 1 illustrates a flowchart of an SSE2 128-bit UTF-32 source to UTF-16 target process in accordance with one or more of various embodiments. [Figure 27] 1 illustrates a flowchart for a process for AVX-512 512-bit UTF-32 source to UTF-16 target in accordance with one or more of various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] Various embodiments will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific exemplary embodiments in which the invention may be practiced. However, embodiments may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the embodiments to those skilled in the art. Among other things, the various embodiments may be methods, systems, media, or devices. Accordingly, the various embodiments may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Therefore, the following detailed description is not to be construed in a limiting sense.

[0005] Throughout this specification and the claims, the following terms take the meanings expressly associated therewith, unless the context clearly dictates otherwise. As used herein, the phrase "in one embodiment" may refer to the same embodiment, but not necessarily to the same embodiment. Further, as used herein, the phrase "in another embodiment" may refer to different embodiments, but not necessarily to different embodiments. Thus, as described below, various embodiments can be readily combined without departing from the scope or spirit of the invention.

[0006] Additionally, as used herein, the term "or" is an inclusive "or" operator and is equivalent to the term "and / or" unless the context clearly dictates otherwise. The term "based on" is not exclusive and allows for based on additional unlisted factors unless the context clearly dictates otherwise. Furthermore, throughout this specification, the meanings of "a," "an," and "the" include plural references. The meaning of "in" includes "in" and "on."

[0007] For example, with respect to the exemplary embodiments, the following terms are also used herein in accordance with their corresponding meanings unless the context clearly dictates otherwise:

[0008] As used herein, the term "engine" refers to logic embodied in hardware or software instructions that may be written in a programming language, such as C, C++, Objective-C, COBOL, Java™, PHP, Perl, JavaScript, Ruby, VBScript, C#, or other Microsoft .NET™ languages. An engine may be compiled into an executable program or written in an interpreted programming language. Software engines can be called from other engines or from themselves. An engine as described herein refers to one or more logical modules that may be merged with other engines or applications or divided into sub-engines. An engine can be stored in a non-transitory computer-readable medium or computer storage device and stored on and executed by one or more general-purpose computers, thus creating a special-purpose computer configured to provide the engine.

[0009] As used herein, the term "data source" refers to a database, application, service, file system, etc. that stores or provides information for an organization. Examples of data sources can include RDBMS databases, graph databases, spreadsheets, file systems, document management systems, local or remote data streams, etc. In some cases, data sources are organized around one or more tables or table-like structures. In other cases, data sources are organized as graphs or graph-like structures.

[0010] As used herein, the term "data model" refers to one or more data structures that provide a representation of one or more portions of an underlying data source. In some cases, a data model can provide a view of a data source for a particular application. A data model may be considered a view or interface to the underlying data source. In some cases, a data model may map directly to a data source (e.g., effectively a logical pass-through). And in other cases, a data model may be provided by the data source. Data models enable organizations to organize or present information from data sources in ways that may be more useful, more meaningful (e.g., easier to reason about), more secure, etc.

[0011] As used herein, the term "data object" refers to one or more data structures that comprise a data model. In some cases, a data object may be considered a part of a data model. A data object may represent an individual instance of an item or a class or type of item.

[0012] As used herein, the term "panel" refers to an area within a graphical user interface (GUI) that has a defined geometry (e.g., x, y, z order) within the GUI. A panel may be configured to display information to a user or to host one or more interactive controls. The geometry or style associated with a panel may be defined using configuration information, including dynamic rules. Also, in some cases, a user may be enabled to perform actions on one or more panels, such as moving, showing, hiding, resizing, reordering, etc.

[0013] As used herein, the term "visualization model" refers to one or more data structures that represent one or more representations of a data model that may be suitable for use in a visualization displayed on one or more hardware displays. A visualization model may define styling or user interface functionality that may be made available to non-authoring users.

[0014] As used herein, the term "display object" refers to one or more data structures that comprise a visualization model. In some cases, a display object may be considered a portion of a visualization model. A display object may represent an individual instance of an item or an entire class or type of item that may be displayed in a visualization. In some embodiments, a display object may be considered or referred to as a view because it provides a view of a portion of a data model.

[0015] As used herein, the terms "character encoding," "character encoding scheme," or "coded character" refer to computer-readable information that can be used to represent characters for use in display or information processing. Character encoding schemes can be used to map characters from one or more human languages ​​to computer-readable values. Character encodings can also be used to map symbols, glyphs, emojis, etc. to computer-readable values. A standardized character encoding allows different computer applications or services to map computer-readable values ​​to the same / correct characters. For example, Unicode is a character encoding standard that is included in many computer applications to allow characters to be represented consistently across different applications or services.

[0016] As used herein, the term "code point" refers to the encoded value of a particular character for a given character encoding. For example, a code point in a character encoding scheme can refer to a particular character, glyph, or symbol.

[0017] As used herein, the term "character code" refers to an encoding scheme for encoding code points for a character encoding scheme. For example, UTF-8, UTF-16, and UTF-32 are character codes for representing Unicode code points.

[0018] As used herein, the term "code unit" refers to the smallest-sized word or element of a character code used to represent a code point in a character encoding scheme. For example, UTF-8 has 8-bit code units, UTF-16 has 16-bit code units, and UTF-32 has 32-bit code units.

[0019] As used herein, the term "encoding model" refers to a data structure that contains or defines rules, instructions, or code that perform operations to convert characters encoded in one or more character codes into characters encoded in one or more other character codes. An encoding model may include one or more signatures, patterns, instructions, conditions, etc. for distinguishing one character code from another. Similarly, an encoding model may include signatures, patterns, instructions, conditions, etc. for verifying or validating a character code. Thus, as new or additional character codes are anticipated, those skilled in the art may provide encoding models that support the new or additional character codes.

[0020] As used herein, the terms "processor features" and "parallel processing features" refer to specialized instructions, registers, APIs, etc. that enable parallel processing. In some cases, a processor may have more or fewer processor features. In some cases, different processors may have different processor features. Similarly, in some cases, some processors may not have processor features that provide support for parallel processing.

[0021] As used herein, the term "configuration information" refers to information that may include rule-based policies, pattern matching, scripts (e.g., computer-readable instructions), etc., that may be provided from a variety of sources, including configuration files, databases, user input, built-in defaults, etc., or combinations thereof.

[0022] The following briefly describes embodiments of the invention in order to provide a basic understanding of some aspects of the invention. This brief description is not intended to be an extensive overview. It is not intended to identify key or critical elements or to delineate or otherwise narrow the scope. Its purpose is merely to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0023] Briefly, various embodiments are directed to managing character encoding using one or more networked computers including one or more processors. In one or more of the various embodiments, a plurality of characters, each encoded as one or more code units based on a character code, can be provided, with the one or more code units for each character representing a code point of the character encoding scheme.

[0024] In one or more of the various embodiments, one or more processor functions in one or more processors can be determined such that the one or more processor functions can enable parallel processing of one or more code units simultaneously with one or more processors.

[0025] In one or more of various embodiments, the encoding model may be determined based on a character code, one or more processor features, and a target character code.

[0026] In one or more of various embodiments, one or more processor functions can be used to convert one or more code units into one or more target code units stored in the one or more processor functions based on an encoding model, such that the one or more target code units are based on a target character code, and the one or more target code units encode a code point for each character.

[0027] In one or more of various embodiments, multiple target characters may be provided in the target stream such that each target character may be encoded as one or more target code units.

[0028] In one or more of various embodiments, converting one or more code units into one or more target code units using one or more processor functions may include determining one or more instructions based on an encoding model and executing the one or more instructions to simultaneously convert one or more portions of the one or more code units into portions of the one or more target code units.

[0029] In one or more of various embodiments, converting the one or more code units into one or more target code units may include loading first portions of the one or more code units into a first register, such that each code unit is two bytes wide; loading second portions of the one or more code units into a second register, such that each code unit is two bytes wide and the second register is the same width as the first register; and providing the one or more target code units based on packing the first portions of the one or more code units and the second portions of the one or more code units into a third register, such that each target code unit is one byte wide, the third register being the same width as the first register.

[0030] In one or more of various embodiments, converting the one or more code units into one or more target code units may include loading the one or more code units into registers, such that each code unit is one byte wide and three or more code units are used to encode each code point representing each character; and generating the one or more target code units in the registers based on the encoding model, such that each target code unit is two bytes wide and each target code unit encodes a code point representing each character.

[0031] In one or more of various embodiments, the character encoding is Unicode, such that the character code is one or more of Unicode Transformation Format-8 bit (UTF-8), Unicode Transformation Format-16 bit (UTF-16), or Unicode Transformation Format-32 bit (UTF-32), and the target character code is one or more of UTF-8, UTF-16, or UTF-32.

[0032] Example Operating Environment 1 illustrates components of one embodiment of an environment in which embodiments of the present invention may be implemented. Not all of the components are required to practice the present invention, and variations in the arrangement and type of components may be made without departing from the spirit or scope of the present invention. As shown, system 100 of FIG. 1 includes a local area network (LAN) / wide area network (WAN) 110, a wireless network 108, client computers 102-105, a visualization server computer 116, a data source server computer 118, etc.

[0033] At least one embodiment of client computers 102-105 is described in more detail below in connection with FIG. 2. In one embodiment, at least some of client computers 102-105 can operate on one or more wired or wireless networks, such as network 108 or 110. In general, client computers 102-105 can include virtually any computer capable of communicating over a network to send and receive information, perform various online activities, offline actions, and the like. In one embodiment, one or more of client computers 102-105 can be configured to operate within a business or other entity to perform various services for the business or other entity. For example, client computers 102-105 may be configured to operate as a web server, a firewall, a client application, a media player, a mobile phone, a game console, a desktop computer, and the like. However, client computers 102-105 are not limited to these services and may also be used, for example, in connection with end-user computing in other embodiments. It should be appreciated that more or fewer client computers (as shown in FIG. 1) may be included in a system as described herein, and thus embodiments are not constrained by the number or type of client computers employed.

[0034] Computers capable of operating as client computers 102 can include computers that typically connect using wired or wireless communication media, such as personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, and network PCs. In some embodiments, client computers 102-105 can include virtually any portable computer capable of connecting to and receiving information from another computer, such as a laptop computer 103, a mobile computer 104, or a tablet computer 105. However, portable computers are not so limited and can also include other portable computers, such as cellular telephones, display pagers, radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), handheld computers, wearable computers, and integrated devices that combine one or more of the foregoing computers. Thus, client computers 102-105 typically range widely in terms of capabilities and functionality. Furthermore, client computers 102-105 can access a variety of computing applications, including browsers or other web-based applications.

[0035] A web-enabled client computer may include a browser application configured to send requests and receive responses over the web. The browser application may be configured to receive and display graphics, text, multimedia, and the like using virtually any web-based language. In one embodiment, the browser application may display and send messages using JavaScript, Hypertext Markup Language (HTML), Extensible Markup Language (XML), JavaScript Object Notation (JSON), Cascading Style Sheets (CSS), and the like, or a combination thereof. In one embodiment, a user of a client computer may use the browser application to perform various activities over a network (online). However, different applications may also be used to perform various online activities.

[0036] The client computers 102-105 may also include at least one other client application configured to receive or send content to or from another computer. A client application may include the ability to send or receive content, etc. A client application may further provide information that identifies itself, including its type, capabilities, name, etc. In one embodiment, the client computers 102-105 may uniquely identify themselves via any of a variety of mechanisms, including an Internet Protocol (IP) address, a phone number, a mobile identification number (MIN), an electronic serial number (ESN), a client certificate, or other device identifier. Such information may be provided in one or more network packets transmitted between other client computers, the visualization server computer 116, the data source server computer 118, or other computers.

[0037] The client computers 102-105 may further be configured to include a client application that allows an end user to log in to an end user account that may be managed by another computer, such as the visualization server computer 116 or the data source server computer 118. Such an end user account may be configured to allow the end user to manage one or more online activities, including, but not limited to, project management, software development, systems administration, configuration management, search activity, social networking activity, browsing various websites, communicating with other users, etc. The client computers may also be configured to allow the user to view reports, interactive user interfaces, or results provided by the visualization server computer 116 or the data source server computer 118.

[0038] Wireless network 108 is configured to couple client computers 103-105 and their components to network 110. Wireless network 108 may include any of a variety of wireless sub-networks that may further overlay standalone ad-hoc networks, etc., to provide infrastructure-oriented connectivity for client computers 103-105. Such sub-networks may include mesh networks, wireless local area network (WLAN) networks, cellular networks, etc. In one embodiment, the system may include one or more wireless networks.

[0039] The wireless network 108 may further include an autonomous system of terminals, gateways, routers, etc., connected by wireless radio links, etc. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, so that the topology of the wireless network 108 may change rapidly.

[0040] The wireless network 108 may further employ multiple access technologies, including second (2G), third (3G), fourth (4G), and fifth (5G) generation wireless access for cellular systems, WLANs, wireless router (WR) mesh, etc. Access technologies such as 2G, 3G, 4G, 5G, and future access networks may enable wide area coverage for mobile computers, such as the client computers 103-105, with various degrees of mobility. In one non-limiting example, the wireless network 108 may enable wireless connectivity via wireless network access such as Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), etc. In essence, wireless network 108 can include virtually any wireless communication mechanism by which information can travel between client computers 103-105 and another computer, network, cloud-based network, cloud instance, etc.

[0041] Network 110 is configured to couple network computers with other computers, including visualization server computer 116, data source server computer 118, client computer 102, and client computers 103-105, such as via wireless network 108. Network 110 can use any form of computer-readable medium for communicating information from one electronic device to another. Network 110 can also include the Internet, as well as a local area network (LAN), a wide area network (WAN), a direct connection such as via a universal serial bus (USB) port, an Ethernet port, or another form of computer-readable medium, or any combination thereof. On an interconnected set of LANs, including those based on different architectures and protocols, a router acts as a link between the LANs, allowing messages to be sent from one to another. Furthermore, while communication links within a LAN typically include twisted-pair or coaxial cable, communication links between networks may utilize other carrier mechanisms, including analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, or wireless links including, for example, E-carrier, Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), satellite links, or other communication links known to those skilled in the art. Furthermore, communication links may employ any of a variety of digital signaling technologies, including, but not limited to, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, etc. Furthermore, remote computers and other associated electronic devices may be remotely connected to either the LAN or WAN via modems and temporary telephone links. In one embodiment, network 110 may be configured to transport information in the Internet Protocol (IP).

[0042] Additionally, communication media typically embodies computer-readable instructions, data structures, program modules, or other transport mechanisms and includes any information non-transitory or transitory distribution media. By way of example, communication media includes wired media such as twisted pair, coaxial cable, fiber optics, wave guides, and other wired media, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0043] Additionally, one embodiment of the visualization server computer 116, the data source server computer 118, etc., is described in more detail below in connection with FIG. 3. While FIG. 1 depicts the visualization server computer 116, the data source server computer 118, etc., as each being a single computer, the invention or embodiments are not so limited. For example, one or more functions of the visualization server computer 116, the data source server computer 118, etc., may be distributed across one or more separate network computers. Furthermore, in one or more embodiments, the visualization server computer 116, the data source server computer 118, etc., may be implemented using multiple network computers. Furthermore, in one or more of various embodiments, the visualization server computer 116, the data source server computer 118, etc., may be implemented using one or more cloud instances in one or more cloud networks. Thus, these inventions and embodiments should not be construed as limited to a single environment, and other configurations and architectures are contemplated.

[0044] Exemplary Client Computer 2 illustrates an embodiment of a client computer 200, which may include more or fewer components than those illustrated. Client computer 200 may represent, for example, one or more embodiments of the mobile computer or client computer illustrated in FIG.

[0045] Client computer 200 may include a processor 202 in communication with memory 204 via bus 228. Client computer 200 may also include a power supply 230, a network interface 232, an audio interface 256, a display 250, a keypad 252, an illuminator 254, a video interface 242, an input / output interface 238, a haptic interface 264, a global positioning system (GPS) receiver 258, an open-air gesture interface 260, a temperature interface 262, a camera 240, a projector 246, a pointing device interface 266, processor-readable permanent storage 234, and a processor-readable removable storage 236. Client computer 200 may optionally communicate with a base station (not shown) or directly with another computer. Also, in one embodiment, a gyroscope (not shown) may be used within client computer 200 to measure or maintain the orientation of client computer 200.

[0046] Power supply 230 can provide power to client computer 200. Rechargeable or non-rechargeable batteries may be used to provide power. Power may also be provided by an external power source, such as an AC adapter or a powered docking cradle that replenishes or recharges the battery.

[0047] The network interface 232 includes circuitry for coupling the client computer 200 to one or more networks and is configured for use with one or more communications protocols and technologies, including, but not limited to, protocols and technologies implementing any portion of the OSI model for mobile communications (GSM), CDMA, time division multiple access (TDMA), UDP, TCP / IP, SMS, MMS, GPRS, WAP, UWB, WiMAX, SIP / RTP, GPRS, EDGE, WCDMA, LTE, UMTS, OFDM, CDMA2000, EV-DO, HSDPA, or any of a variety of other wireless communications protocols. The network interface 232 is sometimes known as a transceiver, a transceiver, or a network interface card (NIC).

[0048] Audio interface 256 may be configured to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 256 may be coupled to a speaker and microphone (not shown) to enable electronic communication with others or to generate audio acknowledgments for some actions. The microphone in audio interface 256 may also be used for input to or control of client computer 200, such as using voice recognition and detecting touch based on sound.

[0049] Display 250 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), organic LED (OLED), or any other type of light reflective or light transmissive display that can be used with a computer. Display 250 may also include a touch interface 244 configured to receive input from an object such as a stylus or a finger from a human hand, and may sense touch or gestures using resistive, capacitive, surface acoustic wave (SAW), infrared, radar, or other technology.

[0050] Projector 246 may be a remote handheld projector or an integrated projector capable of projecting an image onto any other reflective object, such as a remote wall or screen.

[0051] Video interface 242 may be configured to capture video images, such as still photographs, video segments, infrared video, etc. For example, video interface 242 may be coupled to a digital video camera, a webcam, etc. Video interface 242 may include a lens, an image sensor, and other electronics. The image sensor may include a complementary metal-oxide semiconductor (CMOS) integrated circuit, a charge-coupled device (CCD), or any other integrated circuit for sensing light.

[0052] Keypad 252 may comprise any input device configured to receive input from a user. For example, keypad 252 may include a touch-sensitive numeric dial or keyboard. Keypad 252 may also include command buttons associated with selecting and sending images.

[0053] Illuminator 254 may provide a status indication or may provide light. Illuminator 254 may remain active for a specific period of time or in response to an event message. For example, when active, illuminator 254 may backlight buttons on keypad 252 and remain on while the client computer is powered. Illuminator 254 may also backlight these buttons in various patterns when certain actions are performed, such as dialing another client computer. Illuminator 254 may also illuminate light sources located within the client computer's transparent or translucent case in response to an action.

[0054] Additionally, client computer 200 may also include a hardware security module (HSM) 268 that provides additional tamper-resistant safeguards for generating, storing, or using security / cryptographic information, such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some embodiments, the hardware security module may be employed to support one or more standard public key infrastructures (PKIs) and may be employed to generate, manage, or store key pairs. In some embodiments, HSM 268 may be a standalone computer; in other cases, HSM 268 may be configured as a hardware card that can be added to a client computer.

[0055] Client computer 200 may also include input / output interface 238 for communicating with external peripheral devices or other computers, such as other client computers and network computers. Peripherals may include audio headsets, virtual reality headsets, display screen glasses, remote speaker systems, remote speaker and microphone systems, etc. Input / output interface 238 may utilize one or more technologies, such as Universal Serial Bus (USB), infrared, WiFi, WiMAX, Bluetooth™, etc.

[0056] The input / output interface 238 may also include one or more sensors for determining geographic location information (e.g., GPS), for monitoring power conditions (e.g., voltage sensor, current sensor, frequency sensor, etc.), for monitoring weather (e.g., thermostat, barometer, anemometer, humidity detector, precipitation scale, etc.), etc. The sensors may be one or more hardware sensors that collect or measure data external to the client computer 200.

[0057] Haptic interface 264 may be configured to provide haptic feedback to the user of the client computer. For example, haptic interface 264 may be used to vibrate client computer 200 in a particular manner when another user of the computer is making a phone call. Temperature interface 262 may be used to provide temperature measurement input or temperature change output to the user of client computer 200. Open-air gesture interface 260 may sense the physical gestures of the user of client computer 200 by using, for example, a single or stereo video camera, radar, a gyroscope sensor in a computer held or worn by the user, or the like. Camera 240 may be used to track the physical eye movements of the user of client computer 200.

[0058] The GPS transceiver 258 can determine the physical coordinates of the client computer 200 on the surface of the Earth, which typically outputs a location as latitude and longitude values. The GPS transceiver 258 can also use other geographic positioning mechanisms, including, but not limited to, triangulation, Assisted GPS (AGPS), Extended Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Extended Timing Advance (ETA), Base Station Subsystem (BSS), etc., to further determine the physical location of the client computer 200 on the surface of the Earth. It is understood that under different conditions, the GPS transceiver 258 can determine the physical location of the client computer 200. However, in one or more embodiments, the client computer 200, via other components, can provide other information that can be used to determine the physical location of the client computer, including, for example, a Media Access Control (MAC) address, an IP address, etc.

[0059] In at least one of various embodiments, applications such as operating system 206, other client apps 224, and web browser 226 may be configured to use geographic location information to select one or more location-specific features, such as a time zone, language, currency, calendar formatting, etc. The location-specific features may be used in display objects, data models, data objects, user interfaces, reports, and internal processes or databases. In at least one of various embodiments, the geographic location information used to select the location information may be provided by GPS 258. Also, in some embodiments, the geographic location information may include information provided using one or more geographic location protocols over a network, such as wireless network 108 or network 111.

[0060] A human interface component may be a peripheral device physically separate from client computer 200, enabling remote input or output to client computer 200. For example, information routed as described herein through a human interface component such as display 250 or keyboard 252 may instead be routed via network interface 232 to an appropriate remotely located human interface component. Examples of human interface peripheral components that may be remote include, but are not limited to, audio devices, pointing devices, keypads, displays, cameras, projectors, etc. These peripheral components may communicate via pico networks such as Bluetooth™, Zigbee™, etc. One non-limiting example of a client computer having such a peripheral human interface component is a wearable computer, which may include a remote pico projector along with one or more cameras that remotely communicate with a separately located client computer to sense a user's gestures toward a portion of an image projected by the pico projector onto a reflective surface such as a wall or the user's hand.

[0061] The client computer may include a web browser application 226 configured to send and receive web pages, web-based messages, graphics, text, multimedia, etc. The browser application on the client computer may use virtually any programming language, including Wireless Application Protocol messages (WAP), etc. In one or more embodiments, the browser application may use Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), HTML5, etc.

[0062] Memory 204 may include RAM, ROM, or other types of memory. Memory 204 represents an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 204 may store BIOS 208 for controlling the low-level operation of client computer 200. Memory may also store operating system 206 for controlling the operation of client computer 200. It will be appreciated that this component may include a general-purpose operating system, such as a version of UNIX or Linux™, or a dedicated client computer communications operating system, such as the Windows Phone™ or Symbian™ operating systems. The operating system may include or interface with a Java Virtual Machine module, which enables control of hardware components or operating system operation via Java application programs.

[0063] Memory 204 may further include one or more data storage devices 210 that may be utilized by client computer 200 to store, among other things, applications 220 or other data. For example, data storage device 210 may also be used to store information describing various capabilities of client computer 200. The information may then be provided to another device or computer in any of a variety of ways, including as part of a header during a communication, upon request, etc. Data storage device 210 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, etc. Data storage device 210 may further include program code, data, algorithms, etc. used by a processor, such as processor 202, to execute and implement actions. In one embodiment, at least some of data storage device 210 may be stored on another component of client computer 200, including, but not limited to, non-transitory processor-readable removable storage device 236, processor-readable non-removable storage device 234, or even external to the client computer.

[0064] Applications 220 may include computer-executable instructions that, when executed by client computer 200, send, receive, or otherwise process instructions and data. Applications 220 may include, for example, a client visualization engine 222, other client applications 224, a web browser 226, etc. The client computer may be configured to exchange communications with one or more servers.

[0065] Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, visualization applications, and the like.

[0066] Additionally, in one or more embodiments (not shown), client computer 200 may include, instead of a CPU, an embedded logic hardware device such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or the like, or a combination thereof. The embedded logic hardware device may directly execute its embedded logic to perform actions. Also, in one or more embodiments (not shown), client computer 200 may include, instead of a CPU, one or more hardware microcontrollers. In one or more embodiments, the one or more microcontrollers may directly execute their embedded logic to perform actions and access their own internal memory and their own external input / output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as a system on a chip (SOC).

[0067] Exemplary Network Computer 3 illustrates one embodiment of a network computer 300 that may be included in a system implementing one or more of the various embodiments. The network computer 300 may include more or fewer components than those illustrated in FIG. 3. However, the illustrated components are sufficient to disclose an exemplary embodiment for practicing these present inventions. The network computer 300 may represent, for example, an embodiment of at least one of the visualization server computer 116, the data source server computer 118, etc. of FIG. 1.

[0068] A network computer, such as network computer 300, can include a processor 302 that can communicate with memory 304 via bus 328. In some embodiments, processor 302 may be comprised of one or more hardware processors or one or more processor cores. In some cases, one or more of the one or more processors may be specialized processors designed to perform one or more specialized actions, such as those described herein. Additionally, in some embodiments, a processor, such as processor 302, can support or provide one or more processing functions that enable parallel processing of data of various sizes or data types. Network computer 300 also includes a power supply 330, a network interface 332, an audio interface 356, a display 350, a keyboard 352, an input / output interface 338, a processor-readable persistent storage 334, and a processor-readable removable storage 336. Power supply 330 provides power to network computer 300.

[0069] The network interface 332 includes circuitry for coupling the network computer 300 to one or more networks and is configured for use with one or more communication protocols and technologies, including, but not limited to, protocols and technologies implementing any portion of the Open Systems Interconnection Model (OSI model), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), General Packet Radio Service (GPRS), WAP, Ultra Wideband (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMAX), Session Initiation Protocol / Real-time Transport Protocol (SIP / RTP), or any of a variety of other wired and wireless communication protocols. The network interface 332 is sometimes known as a transceiver, a transceiver, or a network interface card (NIC). Network computer 300 may optionally communicate with a base station (not shown) or directly with another computer.

[0070] Audio interface 356 is configured to generate and receive audio signals, such as the sound of a human voice. For example, audio interface 356 may be coupled to a speaker and microphone (not shown) to enable electronic communication with others or to generate audio acknowledgments for some actions. The microphone in audio interface 356 may also be used for input to or control of network computer 300, for example, using voice recognition.

[0071] Display 350 may be a liquid crystal display (LCD), gas plasma, electronic ink, light emitting diode (LED), organic LED (OLED), or any other type of light reflective or light transmissive display that can be used with a computer. In some embodiments, display 350 may be a handheld projector or picoprojector that can project an image onto a wall or other object.

[0072] Network computer 300 may also include an input / output interface 338 for communicating with external devices or computers not shown in Figure 3. Input / output interface 338 may utilize one or more wired or wireless communication technologies, such as USB™, Firewire™, WiFi, WiMAX, Thunderbolt™, infrared, Bluetooth™, Zigbee™, serial port, parallel port, etc.

[0073] Input / output interface 338 may also include one or more sensors for determining geographic location information (e.g., GPS), monitoring power conditions (e.g., voltage sensor, current sensor, frequency sensor, etc.), monitoring weather (e.g., thermostat, barometer, anemometer, humidity detector, precipitation scale, etc.), etc. The sensors may be one or more hardware sensors that collect or measure data external to network computer 300. Human interface components may be physically separate from network computer 300, allowing for remote input or output to network computer 300. For example, information routed as described herein through a human interface component such as display 350 or keyboard 352 may instead be routed via network interface 332 to an appropriate human interface component located elsewhere on the network. Human interface components include any component that enables a computer to receive input from or send output to a human user of the computer. Thus, a pointing device such as a mouse, stylus, or trackball can communicate via pointing device interface 358 to receive user input.

[0074] The GPS transceiver 340 can determine the physical coordinates of the network computer 300 on the Earth's surface, typically outputting a location as latitude and longitude values. The GPS transceiver 340 can also use other geographic positioning mechanisms, including, but not limited to, triangulation, Assisted GPS (AGPS), Extended Observed Time Difference (E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), Extended Timing Advance (ETA), Base Station Subsystem (BSS), etc., to further determine the network computer 300's physical location on the Earth's surface. It is understood that under different conditions, the GPS transceiver 340 can determine the network computer 300's physical location. However, in one or more embodiments, the network computer 300, via other components, can provide other information that can be used to determine the client computer's physical location, including, for example, a media access control (MAC) address, an IP address, etc.

[0075] In at least one of various embodiments, applications such as operating system 306, coding engine 322, visualization engine 324, modeling engine 326, and other applications 329 may be configured to use the geographic location information to select one or more location-specific features, such as time zone, language, currency, currency formatting, calendar formatting, etc. The location-specific features may be used in user interfaces, dashboards, visualizations, reports, and internal processes or databases. In at least one of various embodiments, the geographic location information used to select the location information may be provided by GPS 340. Also, in some embodiments, the geographic location information may include information provided using one or more geographic location protocols over a network, such as wireless network 108 or network 111.

[0076] Memory 304 may include random access memory (RAM), read-only memory (ROM), or other types of memory. Memory 304 represents an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Memory 304 stores a basic input / output system (BIOS) 308 for controlling the low-level operation of network computer 300. Memory also stores an operating system 306 for controlling the operation of network computer 300. It will be understood that this component may include a general-purpose operating system, such as a version of UNIX or Linux, or a dedicated operating system, such as Microsoft Corporation's Windows operating system or Apple Corporation's OSX operating system. The operating system may include or interface with one or more virtual machine modules, such as a Java Virtual Machine module, which enables control of hardware components or operating system operation via Java application programs. Other runtime environments may also be included.

[0077] Memory 304 may further include one or more data storage devices 310 that may be utilized by network computer 300 to store, among other things, applications 320 or other data. For example, data storage device 310 may also be used to store information describing various capabilities of network computer 300. The information may then be provided to another device or computer in any of a variety of ways, including as part of a header during a communication, upon request, etc. Data storage device 310 may also be used to store social networking information, including address books, buddy lists, aliases, user profile information, etc. Data storage device 310 may further include program code, data, algorithms, etc. used by a processor, such as processor 302, to execute and perform actions, such as those described below. In one embodiment, at least a portion of data storage device 310 may be stored on another component of network computer 300, including, but not limited to, non-transitory media in processor-readable removable storage device 336, processor-readable non-transitory storage device 334, or any other computer-readable storage device within or external to network computer 300. The data store 310 may include, for example, a data model 314, a data source 316, a visualization model 318, an evaluation model 319, and the like.

[0078] Applications 320 may include computer-executable instructions that, when executed by network computer 300, send, receive, or otherwise process messages (e.g., SMS, multimedia messaging service (MMS), instant messages (IM), email, or other messages), audio, video, and enable electronic communication with another user of another mobile computer. Other examples of application programs include calendars, search programs, email client applications, IM applications, SMS applications, Voice over Internet Protocol (VOIP) applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, etc. Applications 320 may include an encoding engine 322, a visualization engine 324, a modeling engine 326, other applications 329, etc., which may be configured to perform the actions of the embodiments described below. In one or more of various embodiments, one or more of the applications may be implemented as a module or component of another application. Further, in one or more of various embodiments, an application may be implemented as an operating system extension, module, plug-in, etc.

[0079] Additionally, in one or more of various embodiments, the coding engine 322, the visualization engine 324, the modeling engine 326, the other applications 329, etc., can operate in a cloud-based computing environment. In one or more of various embodiments, these and other applications that comprise the management platform can run within virtual machines or virtual servers that can be managed in the cloud-based computing environment. In one or more of various embodiments, in this context, applications can progress from one physical network computer to another within the cloud-based environment, depending on performance and scaling considerations that are automatically managed by the cloud computing environment. Similarly, in one or more of various embodiments, virtual machines or virtual servers dedicated to the coding engine 322, the visualization engine 324, the modeling engine 326, the other applications 329, etc., can be automatically provisioned and decommissioned.

[0080] Also, in one or more of various embodiments, the coding engine 322, visualization engine 324, modeling engine 326, other applications 329, etc. may be located on virtual servers running in a cloud-based computing environment rather than being tied to one or more particular physical network computers.

[0081] Additionally, network computer 300 may also include hardware security module (HSM) 360 that provides additional tamper-resistant safeguards for generating, storing, or using security / cryptographic information, such as keys, digital certificates, passwords, passphrases, two-factor authentication information, etc. In some embodiments, hardware security module may be employed to support one or more standard public key infrastructures (PKIs) and may be employed to generate, manage, or store key pairs. In some embodiments, HSM 360 may be a stand-alone network computer; in other cases, HSM 360 may be configured as a hardware card that can be installed in the network computer.

[0082] Additionally, in one or more embodiments (not shown), the network computer 300 may include, instead of a CPU, an embedded logic hardware device such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or the like, or a combination thereof. The embedded logic hardware device may directly execute its embedded logic to perform actions. Also, in one or more embodiments (not shown), the network computer may include, instead of a CPU, one or more hardware microcontrollers. In one or more embodiments, the one or more microcontrollers may directly execute their embedded logic to perform actions and access their own internal memory and their own external input / output interfaces (e.g., hardware pins or wireless transceivers) to perform actions, such as a system on a chip (SOC).

[0083] Exemplary Logical System Architecture 4 illustrates a logical architecture of a system 400 for encoding variable-length characters using concurrent processing, according to one or more of various embodiments. In one or more of various embodiments, system 400 may be comprised of various components, including one or more modeling engines, such as modeling engine 402, one or more visualization engines, such as visualization engine 404, one or more visualizations, such as visualization 406, one or more data sources, such as data source 410, one or more visualization models, such as visualization model 408, one or more encoding engines, such as encoding engine 412, or one or more encoding models, such as encoding model 414.

[0084] In one or more of various embodiments, the modeling engine 402 can be configured to allow a user to design one or more visualization models, which can be provided to the visualization engine 404. Thus, in one or more of various embodiments, the visualization engine 404 can be configured to generate one or more visualizations based on the visualization models.

[0085] In one or more of the various embodiments, the modeling engine can be configured to access one or more data sources, such as data source 410. In some embodiments, the modeling engine can be configured to include a user interface that allows a user to browse various data source information, data objects, etc. to design visualization models that can be used to generate visualizations of information stored in the data sources.

[0086] Thus, in some embodiments, a visualization model may be designed to provide visualizations that include charts, plots, graphs, tables, graphics, styling, narrative text, interactive elements, user interface features, etc. In some embodiments, a user may be provided with a graphical user interface that allows them to interactively design the visualization model such that various elements or display objects within the visualization model may be associated with data from one or more data sources, such as data source 410.

[0087] In one or more of various embodiments, a data source, such as data source 410, may include one or more of a database, data store, file system, etc., which may be located locally or remotely. In some embodiments, the data source may be provided by another service over a network. In some embodiments, there may be one or more components (not shown) that filter or otherwise provide an administrative view or access to the data in the data source.

[0088] In one or more of various embodiments, the visualization models may be stored in one or more data stores, such as visualization model storage 408. In this example, for some embodiments, visualization model storage 408 represents one or more databases, file systems, etc. for storing, protecting, or indexing the visualization models.

[0089] In one or more of various embodiments, a visualization engine, such as visualization engine 404, can be configured to parse or otherwise interpret visualization models and data from data sources to generate one or more visualizations that can be displayed to a user.

[0090] In one or more of various embodiments, a visualization model, a data model, or a visualization can be configured to include encoded characters that can be used for annotation, labeling, narrative, configuration information, etc. In some embodiments, characters stored in a data model, a visualization model, a data source, etc. can be encoded using a character code that may require decoding or conversion to a different character code before the characters can be displayed on a display device. Thus, in some embodiments, a visualization engine, etc. can be configured to use a coding engine, such as coding engine 412, to encode or decode characters.

[0091] Thus, in some embodiments, a visualization engine or the like may be configured to selectively encode or decode characters from one character encoding to another. In some embodiments, a visualization engine or encoding engine may be configured to use one or more tests or comparisons to determine the character encoding used in a source and determine whether that character encoding may be compatible with one or more targets. For example, in some embodiments, a source or target may include one or more of a data source, a data model, a visualization model, configuration information, a display system, etc.

[0092] In one or more of various embodiments, the encoding engine may be configured to use an encoding model to identify character codes. Similarly, in one or more of various embodiments, the encoding engine may be configured to convert characters from one or more character codes to one or more other character codes using an encoding model. For example, in some embodiments, the visualization platform may be configured to support one or more Unicode character codes, such as UTF-8, UTF-16, UTF-32, etc. However, while the Unicode character code may be conventionally used in many systems, those skilled in the art will understand that other encoding standards or character codes may be used without departing from the scope of the invention disclosed herein. Thus, in some embodiments, one or more encoding models may be provided to account for different character codes, availability of operating system features, availability of processor features, local environment, local requirements, etc.

[0093] In one or more of various embodiments, the encoding model may be determined on the fly based on parameters or arguments provided by configuration information, etc. Also, in some embodiments, one or more encoding models may include rules, instructions, pattern matching, etc. that can test whether a character can be encoded in a character code supported by a given encoding model. For example, in some embodiments, an encoding model for converting UTF-16-encoded ASCII characters to UTF-8 may verify that the first byte of a UTF-16 word is set to 0 and that the second byte of the word is a valid ASCII value to verify that a provided character can be converted by the encoding model.

[0094] In one or more of various embodiments, the encoding model may enable a variety of conventional or custom character codes to be supported, such as by encoding engine 412. Thus, in some embodiments, when new or additional character codes may be introduced, a corresponding encoding model may be provided to support the new or additional characters.

[0095] 5 illustrates a logical schematic diagram of a portion of a system 500 for encoding variable-length characters using concurrent processing, according to one or more of various embodiments. In one or more of various embodiments, the system 500 may include a coding engine 502 for encoding, decoding, or converting characters from one character code to another, a coding model 510 for providing specific rules, instructions, etc. for performing character code validation or conversion, and one or more data stores, such as a data store 504 for storing character maps, code point tables, etc. Furthermore, in this example, a source stream 506 represents characters being provided to the coding engine 502 for encoding or decoding, and a target stream 508 represents characters that may have been converted by the coding engine 502 from one character code to another.

[0096] In one or more of various embodiments, the encoding engine may be configured to receive a stream of source characters, represented here by source stream 506. Accordingly, in some embodiments, the encoding engine may be configured to employ an encoding model to evaluate the incoming characters provided by source stream 506. In one or more of various embodiments, if one or more encoding models support the requested transformation, the encoding engine 502 may be configured to use one or more character codebooks, or the like, which may be stored in data store 504, to map the source characters provided by source stream 506 to target characters, which may be provided in target stream 508.

[0097] In one or more of various embodiments, the encoding engine 502 can be configured to use one or more encoding models to automatically determine the particular character code to be used for a source character or a target character based on one or more rules, instructions, patterns, etc. contained in the encoding model.

[0098] In some embodiments, the encoding engine may be configured to determine available encoding models based on configuration information. For example, in some embodiments, various character codes may define one or more markers or signals within some characters that allow the encoding engine or encoding model to identify different character codes from encoded characters. Alternatively, in some embodiments, character codes for characters may be determined based on a source stream or a target stream. For example, in some embodiments, the source stream may provide characters from one or more documents or document portions that include metadata that can identify character codes of interest.

[0099] In one or more of various embodiments, one or more of the source stream 506 or the target stream 508 can be configured to support a character code that can use a variable number of bytes or words to represent a character. In some embodiments, the character code can be configured to construct a character using one or more code units depending on the character being represented. For example, a character code that supports a limited number of characters, such as ASCII, may require fewer code values ​​than another character code that can support more or different characters, such as a character code that can support a language that includes one or more characters not supported by ASCII.

[0100] Those skilled in the art will be familiar with one or more standard character encoding standards, such as Unicode and its character codes (e.g., UTF-8, UTF-16, and UTF-32). Therefore, for brevity and clarity, some terminology from this well-known character encoding system may be used herein. For example, a code point may be considered to be a specific value that can be used to identify a character in a character codebook similar to the definition provided by Unicode. Similarly, in some embodiments, the term code unit may be used to represent the minimum amount of memory required to represent a character having a particular character code. For example, the well-known variable-length character code UTF-8 is a character code with 8-bit code units, while the well-known variable-length character code UTF-16 is a character code with 16-bit code units.

[0101] In one or more of various embodiments, a variable-length character code can be configured to require additional code units to represent some characters. For example, a character code with 8-bit code units can represent 256 values ​​with one code unit. For example, UTF-8 can represent ASCII values ​​with a single code unit, but requires additional code units to represent non-ASCII characters. Similarly, in some embodiments, UTF-16 has 16-bit code units, allowing a single UTF-16 code unit to represent more characters than a single UTF-8 code unit. However, in some embodiments, code units may be combined into an ordered sequence that allows more characters to be represented. Thus, for example, a UTF-8 character code can represent the same number of characters as UTF-16 or UTF-32 by using more code units. Thus, in some embodiments, one or more code units of a given character code can be combined to represent a code point that can be used to identify a specific character in a required character codebook. For completeness, note that the character encoding UTF-32 also encodes Unicode characters using 32-bit code units, each capable of storing a complete code point that maps to a Unicode character. In contrast, UTF-8 may require up to four code units to represent the single character that one UTF-32 code unit represents. Similarly, UTF-16 may require up to two 16-bit code units to represent the same character as a single UTF-32 code unit.

[0102] In one or more of various embodiments, variable-length character codes can be configured to minimize or reduce the number of code units required to represent some characters. For example, in Unicode, the first 128 code points represent ASCII characters, and therefore, a single UTF-8 code unit of 8 bits (1 byte) can be used to represent a standard ASCII character. In contrast, for example, UTF-16 requires 16 bits (2 bytes) to represent each ASCII character, and UTF-32 requires 32 bits to represent each ASCII character. Also, for example, UTF-8 may require three code units for a Japanese character. UTF-16 requires only one code unit. Therefore, in this situation, UTF-16 may use less memory (2 bytes) than UTF-8 (3 bytes). Therefore, depending on the situation, UTF-8 may or may not use less memory than UTF-16. Also note that UTF-32 never uses less memory than UTF-8 or UTF-16, it always uses more (equal is rarely true, only when all code points have values ​​greater than 16 bits).

[0103] Thus, in some embodiments, it may be advantageous to convert between character codes with different code units for various reasons depending on the application or local environment. For example, using a character code with larger code units allows a larger character set to be encoded in a single code unit. However, in some embodiments, if the code point for the character being encoded can fit into a smaller code unit, using a larger code unit may use more memory than necessary because a larger code unit is not required to represent the character. For example, if all or most of the characters used by the visualization platform are ASCII characters, using a character code with code units larger than 8 bits may be disadvantageous if memory or storage space is an issue. Also, in some embodiments, the visualization platform may be configured to use character codes with larger code units when storing characters if data storage costs are low, such as when storing files on a file system. However, in some embodiments, using a character code with larger code units than necessary may be disadvantageous if the character code encoded in the larger code units is stored in expensive or scarce memory.

[0104] Additionally, in one or more of various embodiments, the cost of transmitting a character stream over one or more networks can be reduced by reducing the amount of memory or number of bytes. For example, transmitting an XML file / stream composed entirely of ASCII characters (which is common in XML) may require half the memory in UTF-8 as in UTF-16, which may be half the cost of UTF-32. Thus, reducing the amount of data transmitted by a factor of two or four may reduce the time it may take to complete the transmission or reduce the cost to complete the transmission.

[0105] Thus, in one or more of various embodiments, a visualization engine or the like may be configured to convert between different character codes using an encoding engine. For example, in some embodiments, a visualization platform may be configured to encode characters stored in a file using UTF-16 and then convert the encoded characters to UTF-8 if they are read into scarce or expensive memory (e.g., RAM) as part of generating a visualization.

[0106] FIG. 6 illustrates a logical schematic diagram of a portion of an encoding engine 600 for encoding variable length characters using concurrent processing according to one or more of various embodiments.

[0107] As described above, in one or more of the various embodiments, the encoding engine may be configured to encode or decode various character codes. In some embodiments, the encoding engine may be configured to convert characters from one or more character codes to one or more other character codes.

[0108] In this example, for some embodiments, character streams 602A and 602B may represent source streams of encoded characters. In some embodiments, character streams 602A and 602B may be considered I / O streams provided by one or more other services or processes (e.g., an operating system, a file system, a network stack, etc.).

[0109] In conventional encoding engines, character streams may be processed using CPU registers that correspond to the word size of the CPU. For example, in some embodiments, the word size for a 32-bit CPU may be 32 bits, or the word size for a 64-bit CPU may be 64 bits. In some embodiments, characters from a character stream may traditionally be processed code unit by code unit. Thus, in some prior art embodiments, code units may be decoded / encoded one code unit at a time. Typically, this requires looping through each individual code unit so that each code unit can be evaluated individually until each code point is decoded / encoded. Thus, overall encoding / decoding performance may be affected by one or more cache misses, jumps, branches, etc. that may be inherent in processing code units individually.

[0110] In contrast, some CPUs may support one or more processor features that may provide a dedicated data path that provides physical or logical width registers that may be larger than the CPU word size. For example, a CPU with a 64-bit word size may provide a wider data path that provides 128-bit registers, referred to herein as width registers, rather than being limited to 64-bit registers. Thus, in this example, if width registers are used, the CPU can process 128 bits in a single operation rather than being limited to processing a single code unit. Furthermore, some CPUs may provide a dedicated API, instruction set, etc. that can support parallel operations on data stored in one or more registers or similar locations within the processor.

[0111] In this example, conventional encoding engine 604 may enable character codes to be processed code-unit by code-unit. Thus, in this example, word 606 represents sequentially processing code units and loading them into a single code point before converting them into a target code unit. In one or more of various embodiments, conventional encoding engine 604 may enable decoding / encoding characters from character stream 602A into characters for character stream 608A.

[0112] In contrast, in some embodiments, the encoding engine 610 is not limited to a traditional CPU data path, but may be configured to employ processor features that may utilize a width data path, physical width registers, logical width registers, etc. For example, in some embodiments, more code units may be loaded into a width register, such as width register 612. Thus, in some embodiments, more code units may be processed simultaneously in parallel. In this example, width register 612 may be considered to have the size / length of four traditional registers (e.g., register 606). Thus, for UTF-8, a 128-bit wide register can process 16 code units in parallel. A 512-bit wide register can process 64 UTF-8 code units in parallel. Also, although UTF-16 and UTF-32 are scaled down by a factor of 2 and 4, respectively, processing UTF-16 or UTF-32 code units may be significantly faster than processing each code unit sequentially. In some embodiments, if the encoding engine 610 uses width registers, such as width register 612, to transform encoded characters, they may be provided to a conventional output stream, such as output stream 608B, similar to a conventional encoding engine.

[0113] Note that in this example, for simplicity and clarity, conventional coding engine 604 and coding engine 610 are shown as using three registers. Those skilled in the art will understand that a CPU may include several registers, including one or more general-purpose registers or one or more special-purpose / limited-purpose registers. However, register 606 and width register 612 are at least sufficient to implement the invention disclosed herein.

[0114] In one or more of various embodiments, the encoding engine can be configured to determine the availability of processor features that support parallel processing. In some embodiments, the encoding engine can use an API, system calls, or configuration information to determine the features supported by a given CPU. Thus, in some embodiments, the encoding engine can be configured to select among different instructions or encoding models based on the capabilities of the CPU. For example, if the CPU does not provide processor features that support parallel operations, the encoding engine can be configured to select an encoding model that uses serial operations rather than parallel operations.

[0115] FIG. 7 illustrates a logical schematic diagram of a table 700 for comparing character codes for encoding variable length characters using concurrent processing according to one or more of various embodiments.

[0116] In one or more of various embodiments, the encoding engine may be configured to encode / decode variable-length characters using simultaneous processing of various character codes, which may support various character encoding schemes, including Unicode, ASCII, ECMA-6, EBCDIC, custom encoding schemes, etc. In this example, for simplicity and clarity, character codes UTF-8, UTF-16, and UTF-32 are shown to encode Unicode characters, but one skilled in the art will understand that the encoding engine may be configured to support other encoding schemes for encoding variable-length characters using simultaneous processing.

[0117] In this example, table 700 includes a column 702 for storing characters, a column 704 indicating character code types, a column 706 indicating code units for characters, and so on. In this example, column 706 is further broken down into individual byte-sized units for different character codes. In this example, column 706 is divided into sub-columns corresponding to code units that can be used to represent characters. In this example, UTF-8 is a character code with 8-bit code units. Therefore, each sub-column represents a code unit. Similarly, for example, UTF-16 is a character code with 16-bit code units, and therefore, two bytes may be required for each code unit. Also, for example, UTF-32 is a character code with 32-bit code units, and therefore, four bytes are required to represent a character using UTF-32.

[0118] In one or more of various embodiments, some character codes can be configured to use a variable number of code units depending on the character being represented. For example, in table 700, character 716 may be represented by a single UTF-8, UTF-16, or UTF-32 code unit, while character 718 may require one or more code units depending on the character code being used. In some embodiments, if the code point for a character can be represented by one code unit, one single code unit may be required. For example, the code point for character 716 may be considered to be 65 (decimal), or 0x01000001 in binary representation; therefore, the code point for this character can be represented using one UTF-8, UTF-16, or UTF-32 code unit because 0x01000001 can fit into one code unit. In contrast, in this example, character 718 may be considered a character with a code point value that is too large to store in an 8-bit code unit. Thus, as shown in this example, representing character 718 in UTF-8 requires one or more code units, but representing character 718 in UTF-16 or UTF-32 requires one code unit.

[0119] However, in some embodiments, as shown here, using a character code to represent a given character in one character code may consume more memory than representing the same character in another character code. For example, representing character 716 using UTF-8 may require 8 bits, while representing character 716 using UTF-16 or UTF-32 may require 16 bits or 32 bits, respectively. Similarly, representing character 718 using UTF-16 may require 16 bits, while using UTF-8 or UTF-32 may require 24 bits or 32 bits, respectively.

[0120] Furthermore, in some embodiments, the conditional loop of sequential code unit processing as well as the directional classification for parallel processing can be selectively optimized, since in some embodiments the 32-bit signature contained in the code unit can be used to classify UTF-8 code unit combinations when converting to a UTF-16 or UTF-32 target to determine the code point group and therefore the number of tail bytes to be associated with the code unit. The classification itself can provide significant speed improvements over current sequential methods, even in the absence of parallel processing. Combining optimized classification with parallel processing provides further improvements in reduced processing time.

[0121] Generalized behavior 8-10 depict generalized operations for encoding variable-length characters using concurrent processing according to one or more of the various embodiments. In one or more of the various embodiments, processes 800, 900, and 1000 described in connection with FIGS. 8-10 may be implemented or performed by one or more processors on a single network computer (or network monitoring computer), such as network computer 300 of FIG. 3. In other embodiments, these processes, or portions thereof, may be implemented by or performed on multiple network computers, such as network computer 300 of FIG. 3. In still other embodiments, these processes, or portions thereof, may be implemented by or performed on one or more virtualized computers, such as those in a cloud-based environment. However, the embodiments are not so limited and various combinations of network computers, client computers, and the like may be utilized. Furthermore, in one or more of the various embodiments, the processes described in connection with FIGS. 8-10 may be used to encode variable-length characters using concurrent processing according to at least one of the various embodiments or architectures, such as those described in connection with FIGS. 4-7. Additionally, in one or more of various embodiments, some or all of the actions performed by processes 800, 900, and 1000 may be partially performed by the encoding engine 322, the visualization engine 324, the modeling engine 326, etc.

[0122] 8 shows a high-level flowchart of a process 800 for encoding variable-length characters using concurrent processing, according to one or more of various embodiments. After a start block, at block 802, in one or more of various embodiments, a source stream and a target stream may be provided to one or more encoding engines.

[0123] In one or more of various embodiments, the source stream can be provided to the encoding engine from a variety of sources. In some embodiments, the source stream can be provided for a file, a database, a network endpoint, a memory buffer, user input, etc. Those skilled in the art will appreciate that in one or more of various embodiments, the source stream can be an input stream that an operating system or the like can provide to the encoding engine, or more generally, to a visualization platform. In many cases, the source stream can be configured to provide an abstraction layer that separates the raw input source that provides the encoded source characters from other services, including the encoding engine.

[0124] Similarly, in some embodiments, the target stream may be an output stream provided by an operating system or the like that provides an abstraction layer that separates the encoding engine from services that may receive or consume the converted characters, such as a file, database, network endpoint, memory buffer, user interface / display, etc.

[0125] At block 804, in one or more of various embodiments, the encoding engine may be configured to determine a source character encoding and a target character encoding. In some embodiments, the source stream or the target stream may be associated with parameters, tags, metadata, etc., that may indicate the source character encoding or the target character encoding. For example, in one or more of various embodiments, if the source stream is a file, the source stream may be associated with metadata that reports the character encoding used to encode its content. Similarly, for example, if the target stream is a user interface display, the user interface may be associated with metadata that may indicate a preferred or supported character encoding. In some embodiments, the encoding engine may be configured to provide one or more APIs that allow a calling service to explicitly declare a source character encoding or request a target character encoding.

[0126] At decision block 806, in one or more of various embodiments, the processor that executes the instructions for the encoding engine may include processing capabilities that allow parallel operation, and if the source stream includes enough code units, control may proceed to block 808. Otherwise, control may proceed to block 816.

[0127] In one or more of various embodiments, processor features for code-unit parallel processing may be supported or provided by some processors (CPUs) and not by others. Also, in some cases, CPUs that support parallel processing features may disable those features. Similarly, in some embodiments, both the CPU and the operating system may be checked to determine whether one or more required parallel processing features can be supported. For example, in some embodiments, if the CPU supports parallel processing features but the operating system does not support parallel processing features, the parallel processing features may be unavailable. In some embodiments, if parallel processing features may be unavailable, the encoding engine may be configured to employ conventional processing features to convert characters.

[0128] In one or more of various embodiments, a CPU may support one or more versions of parallel processing capabilities. Thus, in one or more of various embodiments, older CPUs may support fewer or different processor capabilities than newer CPUs. Similarly, in some embodiments, CPUs from different manufacturers or with different architectures may support different processor capabilities.

[0129] Thus, in some embodiments, the coding engine may be configured to perform one or more tests to verify the presence of sufficient processor functionality, hi some embodiments, the tests to determine available processor functionality may be included in the coding model rather than in the general instructions of the coding engine.

[0130] Also, in some embodiments, different processor capabilities may become available as new CPUs, operating systems, etc. may be provided. Thus, in some embodiments, the encoding engine may be configured to use rules, instructions, conditions, tests, etc. provided via configuration information to test for the presence or extent of processor capabilities.

[0131] Those skilled in the art will recognize particular system calls, feature flags, version IDs, etc. that may correspond to one or more parallel processing features provided by a CPU or enabled by an operating system.

[0132] Additionally, in some embodiments, the encoding engine may be configured to test or otherwise determine whether the number of code units in the source exceeds a minimum value. For example, a source stream may not contain enough code units to employ parallel code unit processing, even though there may be support for parallel processing.

[0133] In one or more of various embodiments, the encoding engine may be configured to determine an encoding model based on the source character code and the target character code at block 808. As described above, the encoding model may be configured to support converting one or more source character codes to one or more target character codes.

[0134] Furthermore, as mentioned above, CPU / operating system support for parallel processing capabilities may vary depending on the manufacturer, model, version, etc. Thus, in some embodiments, two or more encoding models that may support the same character code conversion may support different processor capabilities. For example, a first encoding model and a second encoding model may support the same conversion, but the first encoding model may be limited to supporting lower-performance processor capabilities, while the second encoding model may support newer, more powerful processor capabilities. Thus, in one or more of various embodiments, the encoding engine may be configured to selectively determine one or more encoding models from among those that may support the requested conversion. Thus, in some embodiments, the encoding models may be ranked, etc. In some embodiments, the encoding engine may be configured to employ an encoding model selected from one or more eligible encoding models based on rules, instructions, conditions, etc. provided via configuration information to consider the local environment or requirements.

[0135] At block 810, in one or more of various embodiments, the encoding engine may be configured to use the determined encoding model to convert source characters into characters encoded using the target character code.

[0136] As mentioned above, an encoding model may consist of a data structure, library, etc. that contains specific instructions, rules, conditions, etc. for converting characters from one character code to another.

[0137] Thus, in some embodiments, the encoding engine can be configured to use a selected encoding model to convert source characters in a source character code into characters encoded using a target character code.

[0138] In one or more of various embodiments, an encoding model can be configured to provide instructions that leverage parallel processing capabilities to enable one or more transforms to occur in parallel, reducing branching, looping, and the like. In some embodiments, an encoding model for encoding variable-length characters using concurrent / parallel processing may include loading multiple code units into registers and then employing one or more register operations, such as masking and shifting, that operate directly on the registers, allowing more code units to be processed simultaneously. For example, if a register is 16 bytes wide, 16 UTF-8 code units may be loaded into a single register for processing. In contrast, sequential processing of code units may require processing each code unit individually, such as using a looping deterministic finite automaton. Note that in some embodiments, the instructions or operations defined by a particular encoding model may depend on the source and target character codes involved in the transformation and the availability of processor features. In some embodiments, one or more transforms may be ineligible for parallel processing optimization depending on the availability of processor features.

[0139] At block 812, in one or more of various embodiments, the encoding engine may be configured to provide characters encoded in the target character code to the target stream. In one or more of various embodiments, when the encoding engine completes the conversion operation on the set of characters, the encoding engine may be configured to move the converted code units to conventional registers or memory associated with the target stream. Thus, in some embodiments, other services (e.g., user interfaces, displays, files, databases, etc.) can use the converted characters and receive the characters encoded in the target character code.

[0140] In decision block 814, in one or more of various embodiments, if the source stream has more available characters and the number of code units is sufficient to use parallel conversion, control may loop back to block 810. Otherwise, control may proceed to block 816.

[0141] In one or more of various embodiments, the encoding engine can be configured to process chunks of encoded characters from a source stream. In some embodiments, the number of encoded characters in each chunk can vary depending on one or more of the selected encoding model, the source / target character codes, or available processor features. For example, in some embodiments, if a CPU has processor features that enable 512-bit wide registers, it can process 64 UTF-8 code units at a time or 16 UTF-32 code units at a time. In contrast, a CPU with processor features that provide, for example, 128-bit wide registers can process 16 UTF-8 code units or 4 UTF-32 code units at a time.

[0142] In one or more of the various embodiments, the encoding engine may optionally be configured to convert one or more remaining source characters for which parallel processing capabilities do not exist at block 816. Thus, in some embodiments, where parallel processing capabilities may not be available or the number of source code units is too small for parallel processing, conventional processor capabilities may be used to perform the conversion sequentially for each code unit.

[0143] Note that this block is shown as optional, since in some cases in some embodiments, all source characters may have been processed using parallel processing capabilities.

[0144] Control may then be returned to the calling process in one or more of various embodiments.

[0145] FIG. 9 shows a flowchart of a process 900 for using an encoding model to convert UTF-16 encoded ASCII characters to UTF-8, according to one or more of various embodiments.

[0146] After a start block, in one or more of various embodiments, at decision block 902, if the encoding engine can be used to convert UTF-16 encoded ASCII characters to UTF-8, control can proceed to block 904. Otherwise, control can be returned to the calling process.

[0147] As explained above, in some embodiments, the encoding engine may receive requests to convert encoded characters from one or more character codes to one or more other character codes, etc.

[0148] In this example, the encoding engine may be considered to receive a request to convert UTF-16 encoded characters that may be, or are assumed to be, ASCII characters, which means that they may correspond to Unicode code points 0 through 127. Unicode code points 0 through 127 are mapped to ASCII code points to support compatibility with common ASCII character encodings.

[0149] In one or more of various embodiments, converting ASCII characters encoded as UTF-16 characters may be advantageous because the smallest code unit in UTF-16 is 16 bits (2 bytes). Thus, in this example, 16 bits (2 bytes) are required to represent a character that can be represented using 8 bits (1 byte). Note that in this example, if a non-ASCII code point, such as a Unicode code point with a value greater than 127, is converted from UTF-16 to UTF-8, multiple UTF-8 code units will be required; otherwise, the character will be considered unprintable.

[0150] In one or more of various embodiments, if the source character is not ASCII, the encoding engine can terminate process 900 and therefore evaluate the source character to determine whether a different transformation can be applied.

[0151] At block 904, in one or more of various embodiments, the encoding engine may be configured to determine an encoding model that may be employed to perform the requested transformation.

[0152] In one or more of various embodiments, the encoding engine can be configured to select an encoding model based on the source and target character codes, as well as other factors, such as requiring the target characters to be ASCII-compatible. In this example, for some embodiments, the source character code is UTF-16 and the target character code is UTF-8, with the further requirement or expectation that the source character be converted to a single UTF-8 code unit. Thus, in this example, source characters that cannot be represented using ASCII-compatible code points can be treated as non-printable characters rather than using additional code units to represent them.

[0153] For example, if a user knows that a file stored using UTF-16 encoding contains only or mostly ASCII-compatible characters, the user may request that the characters in the file be converted to UTF-8 to reduce the amount of memory / space used to represent the characters in the file.

[0154] Furthermore, as mentioned above, in some embodiments, the coding model may be selected based on the available parallel processing capabilities provided by the CPU / operating system being used.

[0155] At block 906, in one or more of various embodiments, the encoding engine may be provided with one or more source characters encoded in UTF-16. In one or more of various embodiments, a source stream may be provided that provides characters encoded using UTF-16. Similarly, in some embodiments, a target stream may be provided that may be suitable for storing UTF-8 characters.

[0156] At block 908, in one or more of various embodiments, the encoding engine may be configured to load a first portion of a code unit corresponding to one or more source characters into a first register or other memory location (referred to herein as a register).

[0157] In one or more of various embodiments, the encoding engine can be configured to load as many UTF-16 characters as can fit into a register. For example, in some embodiments, if the processor functions enable a register width of 128 bits (16 bytes), eight 2-byte wide UTF-16 code units can be loaded into a first register.

[0158] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0159] [Table 1]

[0160] In one or more of various embodiments, the encoding engine may be configured to load a second portion of the code unit corresponding to one or more source characters into a second register at block 910. Here, block 910 may be considered similar to block 908, except that the next set of UTF-16 characters may be loaded from the source stream into another register.

[0161] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0162] [Table 2]

[0163] Thus, in this example, the encoding model is configured to read 16 UTF-16 characters, 8 into the first register and 8 into the second register.

[0164] At block 912, in one or more of various embodiments, the encoding engine may be configured to pack the source character code units into a result width register.

[0165] In some embodiments, the encoding engine may be configured to execute a register instruction to pack two-byte UTF-16 code units stored in a first register and a second register into a single result register. In this example, packing the two registers may comprise treating each two bytes in the first and second width registers as a 16-bit integer that can be packed into an 8-bit width result register. In this example, the eight UTF-16 code units in the first register and the eight UTF-16 code units in the second register are treated as 16-bit integers that are packed into the result register as an 8-bit integer. In this example, the register packing function may employ saturation packing, which sets the packed value to the maximum value of the data type if the value held in the 16-bit integer cannot be represented by an 8-bit integer. For example, a 16-bit UTF-16 code unit such as 0x0041 may be packed into an 8-bit integer with a value of 0x41. Also, for example, a 16-bit UTF-16 code unit with a value of 0x0A41 may be packed into an 8-bit integer with a value of 0xFF. This is because the 16-bit value 0x0A41 cannot fit into an 8-bit integer.

[0166] In particular, UTF-16 characters representing ASCII compatible characters have values ​​between 0 and 127 (0x0001 and 0x007F), allowing them to be packed into 8-bit integers without any problems.

[0167] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0168] [Table 3]

[0169] At block 914, in one or more of the various embodiments, the encoding engine may be configured to determine the number of ASCII characters that may be present in the result register.

[0170] In this example, if the result register was generated from a UTF-16 character with an ASCII-compatible code point, the result register now stores 16 UTF-8 code units, each corresponding to an ASCII character. In contrast, if, for some reason, the source stream provided a UTF-16 code unit containing a non-ASCII character, one or more integers in the result register will have the value 0xFF, which is not an ASCII code point.

[0171] In this example, the number of ASCII characters converted is determined by counting the number of 8-bit integers in the result register that are between 0x00 and 0x7F. Also in this example, counting the number of 8-bit integers in the result register that have a value of 0XFF indicates UTF-16 characters from the source stream that are not decomposed into ASCII characters.

[0172] In some embodiments, the parallel processing functionality may support a single operation that generates a mask from a register based on the value (0 or 1) of each most significant bit in an 8-bit integer held in the register. Thus, in one or more of various embodiments, if a result register holds ASCII characters, the MSB of their corresponding 8-bit integers is set to 0. As a final step, the mask generated from the result register may be ORed with itself with 0xFFFF0000 so that the result mask is a 32-bit integer.

[0173] The number of ASCII characters can then be determined based on the position of the least significant bit in the mask.

[0174] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0175] [Table 4]

[0176] At block 916, in one or more of various embodiments, the encoding engine may be configured to provide the transformed characters in the result to the target stream.

[0177] Thus, in some embodiments, the 8-bit UTF-8 code points in the result register may be passed to the calling process as normal (e.g., non-wide register) memory.

[0178] For example, an encoding model configured to support Intel processors may include instructions similar to the following:

[0179] [Table 5]

[0180] In decision block 918, in one or more of various embodiments, if processing should continue, control may loop back to block 906. Otherwise, control may return to the calling process.

[0181] In one or more of various embodiments, there must be sufficient remaining length for both the source and target to continue parallel processing. Otherwise, the parallel construct may reference out of memory, potentially causing an exception fault. Also, in some embodiments, if the first code point is not ASCII, the process should terminate to avoid the possibility of an infinite loop.

[0182] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0183] [Table 6]

[0184] 10 shows a flowchart of a process 1000 for using an encoding model to convert UTF-8 encoded CJK characters to UTF-16 encoded CJK characters, according to one or more of various embodiments. After a start block, at decision block 1002, in one or more of various embodiments, if an encoding engine can be used to convert Unicode CJK characters encoded using UTF-8 to UTF-16 characters, control may proceed to block 1004. Otherwise, control may be returned to the calling process.

[0185] The source character code or the target character code can be determined in a similar manner as described above for process 900, etc.

[0186] In one or more of various embodiments, the encoding engine may be configured to determine an encoding model that may be employed to perform the requested transformation, at block 1004. Similar to the above, the encoding engine may be configured to select an encoding model that may be suitable for transforming the source character code to the target character code.

[0187] At block 1006, in one or more of various embodiments, the encoding engine may be provided with one or more source characters encoded in UTF-8.

[0188] In some embodiments, if the parallel processing functionality associated with the CPU supports 128-bit wide registers, 16 UTF-8 code units may be read into the registers.

[0189] Additionally, in some embodiments, the encoding model may include instructions for generating one or more masks suitable for converting UTF-8 to UTF-16. In this example, the encoding model is designed to convert CJK characters represented using UTF-8. Thus, three UTF-8 code units are required per CJK character. Note that Unicode defines algorithms for converting between UTF-8, UTF-16, or UTF-32. Those skilled in the art will recognize that these may be described in Unicode documents, including at least RFC-3629, RFC-2781, etc. However, the inventions described herein that employ parallel processing capabilities are not disclosed by the Unicode standard or related RFCs.

[0190] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0191] [Table 7]

[0192] At block 1008, in one or more of various embodiments, the encoding engine may be configured to convert head bytes and tail bytes of the source characters.

[0193] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0194] [Table 8]

[0195] At block 1010, in one or more of the various embodiments, the encoding engine may be configured to combine the head bytes and tail bytes in a register.

[0196] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0197] [Table 9]

[0198] At block 1012, in one or more of various embodiments, the encoding engine may be configured to provide the converted UTF-16 code units to a target stream.

[0199] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0200] [Table 10]

[0201] In decision block 1014, in one or more of various embodiments, if processing can continue, control can loop back to block 1006. Otherwise, control can be returned to the calling process.

[0202] Also, in one or more of various embodiments, there must be sufficient remaining length for both the source and target to continue parallel processing. Otherwise, the parallel construct may reference out of memory, potentially causing an exception fault. Also, in some embodiments, if the first code point is not CJK, the process should terminate to avoid a possible infinite loop.

[0203] For example, an encoding model configured to support Intel processors may include one or more instructions similar to the following:

[0204] [Table 11]

[0205] It will be understood that each block in each flowchart diagram, and combinations of blocks in each flowchart diagram, can be implemented by computer program instructions. These program instructions can be provided to a processor to generate a machine, such that the instructions, when executed on the processor, create means for implementing the actions specified in one or more of the flowchart blocks. The computer program instructions, when executed by the processor, can cause the processor to perform a series of operational steps to generate a computer-implemented process, such that the instructions executed thereon provide steps for implementing the actions specified in each flowchart block(s). The computer program instructions can also cause at least some of the operational steps illustrated in each flowchart block to be performed in parallel. Furthermore, some of the steps may be executed across more than one processor, as may occur in a multiprocessor computer system. Furthermore, one or more blocks or combinations of blocks in each flowchart diagram can be executed concurrently with other blocks or combinations of blocks, or in a different order than that illustrated, without departing from the scope or spirit of the present invention.

[0206] Therefore, each block of each flowchart diagram supports a combination of means for performing the specified operations, a combination of steps for performing the specified operations, and program instruction means for performing the specified operations. It should also be understood that each flowchart diagram and combination of blocks within each flowchart diagram can be implemented by a dedicated hardware-based system that performs the specified actions or steps, or a combination of dedicated hardware and computer instructions. The foregoing examples should not be construed as limiting or exhaustive, but rather as illustrative use cases to illustrate at least one implementation of various embodiments of the present invention.

[0207] Additionally, in one or more embodiments (not shown), the logic of the exemplary flowcharts may be executed using an embedded logic hardware device, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or the like, or a combination thereof, instead of a CPU. The embedded logic hardware device may directly execute its embedded logic to perform actions. In one or more embodiments, a microcontroller may be configured to directly execute its own embedded logic to perform actions and access its own internal memory and its own external input and output interfaces (e.g., hardware pins or a wireless transceiver) to perform actions, such as a system on a chip (SOC).

[0208] Exemplary Use Cases 11-27 represent use cases described with respect to a particular character encoding (Unicode) using various processor features that may be available in some processors offered by a particular manufacturer. Note that one skilled in the art will understand that the invention disclosed herein contemplates other character encoding standards or other processor features. Therefore, for the sake of brevity and clarity, additional use cases or examples of other character encoding standards or processors are omitted.

[0209] Figure 11 provides an overview of a portion of a character encoding standard that may be in accordance with one or more of the various embodiments. In this example, the character encoding standard of interest is Unicode. This table provides information about the Unicode character encoding standard to provide context for the Unicode-related examples contained above and below. However, it is assumed that those skilled in the art are familiar with character encoding standards in general, and Unicode in particular.

[0210] In this example, table 1102 represents an informal classification of different portions of the Unicode character map or character codebook. In some cases, character encoding standards may order characters in a codebook based on various characteristics. A character encoding standard may be configured to categorize character encodings such that the encodings may be advantageous given various considerations that may be important to the designer of the character encoding standard.

[0211] In this example, column 1104 represents nicknames, which are informal labels for categories. Note that in this example, column 1104 is described as containing nicknames because the categories or their labels may not be officially included in Unicode. In other cases, such as other character encoding standards, one or more categories according to one or more of various embodiments may be formally declared. Thus, in some cases, column 1104 represents formal category labels.

[0212] In this example, column 1106 represents a range of code points that map to characters in a given category, and in this example, the code points are represented using hexadecimal values.

[0213] In this example, line 1106 declares ASCII nicknames for code points that fall in the range 0x0000-0x007F. Unicode characters supported by the ASCII character encoding are included in this category. Line 1108 declares a Greek category, which represents another range of characters (0x0080-0x07FF). Line 1110 declares a CJK (Chinese, Japanese, and Korean) category, which represents a range of code points that represent many Asian language characters. Additionally, in this example, line 1110 declares an Emoji category, which represents another range of characters (0x0001000-0x0010FFFF), most of which are commonly referred to as Emoji.

[0214] It should be noted that while there may be exceptions or omissions with respect to these informal categories, those skilled in the art will understand that these informal categories provide a useful shortcut for grouping characters into categories with similar characteristics.

[0215] In this example, table 1114 illustrates some of the benefits, limitations, etc. associated with converting from one Unicode character code to another. Column 1116 represents the source character code, column 1118 represents whether character code UTF-8 is the target, column 1120 represents whether character code UTF-16 is the target, and column 1122 represents whether character code UTF-32 is the target. Thus, row 1124 represents the source character code for UTF-8, row 1126 represents the source character code for UTF-16, and row 1128 represents the source character code for UTF-32.

[0216] 12-27 represent generalized operations for encoding variable-length characters using concurrent processing according to one or more of various embodiments. In particular, FIGS. 12-27 may represent actions or operations that may be included in one or more encoding models that may be employed to encode variable-length characters using concurrent processing. Thus, in one or more of various embodiments, processes 1200, 1300, 1400, 1500, 1660, 1700, 1800, 1900, 2000, 2100, 2200, 2300, 2400, 2500, 2600, and 2700 described in connection with FIGS. 12-27 may be implemented or performed by one or more processors on a single network computer (or network monitoring computer), such as network computer 300 of FIG. 3. In other embodiments, these processes, or portions thereof, may be implemented by or performed on multiple network computers, such as network computer 300 of FIG. 3. In still other embodiments, these processes, or portions thereof, may be implemented by or executed on one or more virtualized computers, such as those in a cloud-based environment. However, embodiments are not so limited and various combinations of network computers, client computers, etc. may be utilized. Furthermore, in one or more of various embodiments, the processes described in conjunction with FIGS. 12-27 may be used to encode variable-length characters using concurrent processing according to at least one of various embodiments or architectures, such as those described in conjunction with FIGS. 4-7. Furthermore, in one or more of various embodiments, some or all of the actions performed by processes 1200, 1300, 1400, 1500, 1660, 1700, 1800, 1900, 2000, 2100, 2200, 2300, 2400, 2500, 2600, and 2700 may be partially performed by encoding engine 322, visualization engine 324, modeling engine 326, etc.

[0217] FIG. 12 shows a flowchart of a sequential single code unit UTF-8 source to target (optional) process 1200 according to one or more of various embodiments. After a start block, in one or more of various embodiments, a source stream and a target stream are provided at block 1202. At block 1204, a UTF-16 or UTF-32 target is determined. At decision block 1206, if source code units remain, control may proceed to decision block 1208. Otherwise, control may return to the calling process. At decision block 1208, if target memory may remain, control may proceed to decision block 1208. Otherwise, control may return to the calling process. In one or more of various embodiments, at classification block 1210, the number of tail bytes required from the current head byte code unit is classified. The following action may be selected based on the category of the character(s) or code unit being processed. At block 1212, if the category is ASCII and there are no additional tail bytes, control may proceed to block 1224. In category block 1214, if the category is Greek and there can be one or more additional tail bytes, control may proceed to decision block 1222. In category block 1216, if the category is CJK, there can be two additional tail bytes, so control may proceed to decision block 1222. In category block 1218, if the category is Emoji, there can be three additional tail bytes, so control may proceed to decision block 1222. In category block 1220, if the character is invalid, control may proceed to block 1226. In decision block 1222, if the encoding engine can successfully walk the required tail byte code units, control may proceed to block 1224. Otherwise, control may proceed to block 1226. In block 1224, the encoding engine can provide the converted character to the target stream.In block 1226, the encoding engine may convert the character to a replacement character 0xFFD, which represents an invalid character or a failed conversion.

[0218] FIG. 13 shows a flowchart of a signature 32-bit UTF-8 source to (any) target process 1300 according to one or more of various embodiments. After a start block, block 1302 provides a source stream and a target stream. Block 1304 determines a UTF-16 or UTF-32 target. If, at decision block 1306, four source code units remain, control may proceed to decision block 1308. Otherwise, control may return to another process. If, at decision block 1308, there may be target memory remaining, control may proceed to classification block 1310. Otherwise, control may return to another process. In classification block 1311, classify the code point from the 32-bit signature. If, at category block 1312, the category is ASCII (1 byte), there is no bit manipulation. Control may proceed to block 1324. If, at category block 1314, the category is Greek 2-byte bit manipulation, control may proceed to decision block 1322. In category block 1316, if the category is CJK 3-byte bit manipulation, control may proceed to decision block 1322. In category block 1318, if the category is Emoji 4-byte bit manipulation, control may proceed to decision block 1322. In block 1320, if the character is invalid and a code unit error length is found, control may proceed to block 1326. In block 1322, if the code point value may be overcommitted, control may proceed to block 1326. Control may proceed to block 1324. In block 1324, the converted character may be moved to the target stream. In block 1326, the character is converted to the replacement character 0xFFD.

[0219] FIG. 14 shows a flowchart of an SSE4 128-bit UTF_8 source to target (either) process 1400 in accordance with one or more of various embodiments. At block 1402, provide a source stream and a target stream. At block 1404, determine a UTF-16 or UTF-32 target. At block 1406, are 16 source bytes remaining? At block 1408, is there enough target memory? At block 1410, classify code points using a 32-bit signature. At block 1412, ASCII: 16 potential code points. At block 1414, Greek: 8 potential code points. At block 1416, CJK: 5 potential code points. At block 1418, Emoji: 4 potential code points. At block 1420, find invalid code unit error length. At block 1422, process potential code points using SSE2 and SSE4 SIMD instructions. Advance the source and target according to the number of consecutive related code points found. Block 1424 converts to the replacement character 0xFFD.

[0220] FIG. 15 shows a flowchart of an AVX 512-bit UTF-8 source to (any) target process 1500 in accordance with one or more of various embodiments. At block 1502, provide a source stream and a target stream. At block 1504, determine a UTF-16 or UTF-32 target. At block 1506, are 64 source bytes remaining? At block 1508, is there enough target memory? At block 1510, classify code points using a 32-bit signature. At block 1512, ASCII potential 64 code points. At block 1514, Greek potential 32 code points. At block 1516, CJK potential 21 code points. At block 1518, Emoji potential 16 code points. At block 1520, find invalid code unit error lengths. At block 1522, process potential code points using AVX512 SIMD instructions. Advance the source and target according to the number of consecutive related code points found. Block 1524 converts to the replacement character 0xFFD.

[0221] Figure 16 shows a flowchart of a sequential single code unit (currently) UTF-16 source to UTF-8 target process 1600. Block 1602 provides a source stream and a target stream. Block 1604: Are there any source code units remaining? Block 1606: Remaining target memory? Block 1608: Sort code units based on whole code points or code pairs. Block 1610: ASCII head byte only. Block 1612: Greek head byte and 1 tail byte. Block 1614: CJK head byte and 2 tail bytes. Block 1616: First code pair, second code pair present (1 head byte and 3 tail bytes). Block 1618: Second code pair, invalid without first code pair at beginning. Block 1620: Convert to head and tail bytes. Block 1622: Copy to target stream. Block 1624: Convert to replacement character 0xFFD using 3 bytes.

[0222] FIG. 17 shows a flowchart of an SSE4 128-bit UTF-16 source to UTF-8 target process 1700. At block 1702, provide a source stream and a target stream. At block 1704, are 16 source code units remaining? At block 1706, find remaining target memory? At block 1708, sort code units based on total code points or code pairs. At block 1710, ASCII, 16 potential code points. At block 1712, Greek, 8 potential code points. At block 1714, CJK, 5 potential code points. At block 1716, first code pair, second code pair present. At block 1718, second code pair, invalid without preceding first code pair. At block 1720, Emoji, 4 potential code points. At block 1722, process potential code points using SSE2 and SSE4 SIMD instructions. Advance source and target according to the number of consecutive related code points found. Block 1724 uses three bytes to convert to the replacement character 0xFFD.

[0223] FIG. 18 illustrates a flowchart of an AVX 512-bit UTF-16 source to UTF-8 target process 1800 in accordance with one or more of various embodiments. At block 1802, provide a source stream and a target stream. At block 1804, are 64 source code units remaining? At block 1806, determine remaining target memory? At block 1808, classify code units based on total code points or code pairs. At block 1810, ASCII, 64 potential code points. At block 1812, Greek, 32 potential code points. At block 1814, CJK, 20 potential code points. At block 1816, first code pair, second code pair present. At block 1818, second code pair, invalid without leading first code pair. At block 1820, Emoji, 16 potential code points. At block 1822, process potential code points using AVX512 SIMD instructions. The source and target are advanced according to the number of consecutive associated code points found. At block 1824, three bytes are used to convert to the replacement character 0xFFD.

[0224] Figure 19 shows a flowchart of a sequential single code unit (currently) UTF-32 source to UTF-8 target process 1900. Block 1902 provides a source stream and a target stream. Block 1904: Are there any source code units remaining? Block 1906: Remaining target memory? Block 1908: Sort by one code unit. Block 1910: ASCII, head byte only. Block 1912: Greek, head byte and 1 tail byte. Block 1914: CJK, head byte and 2 tail bytes. Block 1916: Emoji, head byte and 3 tail bytes. Block 1918: Invalid, greater than 0x0010FFFF. Block 1920: Convert to head and tail bytes. Block 1922: Copy to target stream. Block 1924: Convert to replacement character 0xFFD using 3 bytes.

[0225] FIG. 20 illustrates a flowchart of an SSE4 128-bit UTF-32 source to UTF-8 target process 2000 in accordance with one or more of various embodiments. At block 2002, provide a source stream and a target stream. At block 2004, are 16 source code units remaining? At block 2006, determine remaining target memory? At block 2008, sort by one code unit. At block 2010, ASCII, 16 potential code points. At block 2012, Greek, 8 potential code points. At block 2014, CJK, 4 potential code points. At block 2016, Emoji, 4 potential code points. At block 2018, invalid value, greater than 0x0010FFFF. At block 2020, process potential code points using SSE2 and SSE4 SIMD instructions. Advance source and target according to the number of consecutive related code points found. At block 2022, convert to a replacement character 0xFFD using 3 bytes.

[0226] FIG. 21 illustrates a flowchart of an AVX 512-bit UTF-32 source to UTF-8 target process 2100 in accordance with one or more of various embodiments. At block 2102, provide a source stream and a target stream. At block 2104, are 64 source code units remaining? At block 2106, sort by remaining target memory? At block 2108, sort by one code unit. At block 2110, ASCII, 64 potential code points. At block 2112, Greek, 32 potential code points. At block 2114, CJK, 8 potential code points. At block 2116, Emoji, 8 potential code points. At block 2118, invalid value, greater than 0x0010FFFF. At block 2120, process potential code points using SSE2 and SSE4 SIMD instructions. Advance source and target according to the number of consecutive related code points found. At block 2122, convert to replacement character 0xFFD using 3 bytes.

[0227] Figure 22 shows a flowchart of a sequential single code unit (currently) UTF-16 source to UTF-32 target process 2200. At block 2202, provide a source stream and a target stream. At block 2204, are there any source code units remaining? At block 2206, remaining target memory? At block 2208, is it ASCII, Greek, or CJK? At block 2210, is it the first code pair? At block 2212, is the next code unit a second code pair? At block 2214, the second code pair is invalid without the leading first code pair. At block 2216, convert to glyph. At block 2218, convert to replacement character 0xFFD. At block 2220, copy to target stream.

[0228] Figure 23 shows a flowchart of an SSE2 128-bit UTF-16 source to UTF-32 target process 2300. At block 2302, provide a source stream and a target stream. At block 2304, are there 8 source code units? At block 2306, is there enough target memory remaining? At block 2308, perform a single-unit conversion from SSE2 UTF-16 to UTF-32. At block 2310, are there more than 0 code units converted? At block 2312, perform a code-pair conversion from SSE2 UTF-16 to UTF-32. At block 2314, are there more than 0 code points converted? At block 2316, is there a single code unit with no pair? Convert the single code point to the replacement character 0xFFD.

[0229] 24 illustrates a flowchart of an AVX 512-bit UTF-16 source to UTF-32 target process 2400 in accordance with one or more of various embodiments. At block 2402, provide a source stream and a target stream. At block 2404, are there 32 source code units? At block 2406, is there enough target memory remaining? At block 2408, perform a single-unit conversion from AVX512 UTF-16 to UTF-32. At block 2410, are there more than 0 code units converted? At block 2412, perform a code-pair-unit conversion from AVX512 UTF-16 to UTF-32. At block 2414, are there more than 0 code points converted? At block 2416, is there a single code unit with no pair? Convert the single code point to the replacement character 0xFFD.

[0230] Figure 25 shows a flowchart of a sequential single code unit UTF-32 source to UTF-16 target process 2500. At block 2502, provide a source stream and a target stream. At block 2504, are there any source code units remaining? At block 2506, remaining target memory? At block 2508, is there a code unit less than or equal to 0xFFFF? At block 2510, is it a glyph? At block 2512, convert to a code unit pair. At block 2514, convert to replacement character 0xFFD. At block 2516, copy to target stream.

[0231] 26 illustrates a flowchart of an SSE2 128-bit UTF-32 source to UTF-16 target process 2600 in accordance with one or more of various embodiments. At block 2602, provide a source stream and a target stream. At block 2604, are there 8 source code units? At block 2606, is there enough target memory remaining? At block 2608, perform a single-unit conversion from SSE2 UTF-32 to UTF-16. At block 2610, are there more than 0 code units to convert? At block 2612, perform a code-to-unit conversion from SSE2 UTF-32 to UTF-16. At block 2614, are there more than 0 code points converted? At block 2616, convert the single code point to the replacement character 0xFFD.

[0232] Figure 27 shows a flowchart of an AVX512 128-bit UTF-32 source to UTF-16 target process 2700 in accordance with one or more of various embodiments. At block 2702, provide a source stream and a target stream. At block 2704, are there 32 source code units? At block 2706, is there enough target memory remaining? At block 2708, perform a single-unit conversion from AVX512 UTF-32 to UTF-16. At block 2710, are there more than 0 code units converted? At block 2712, perform a code-to-unit conversion from AVX512 UTF-32 to UTF-16. At block 2714, are there more than 0 code points to convert? At block 2716, invalid because it is greater than 0x0010FFFF. Convert the single code point to the replacement character 0xFFD.

[0233] It will be understood that each block in each flowchart diagram, and combinations of blocks in each flowchart diagram, can be implemented by computer program instructions. These program instructions can be provided to a processor to generate a machine, such that the instructions, when executed on the processor, create means for implementing the actions specified in one or more of the flowchart blocks. The computer program instructions, when executed by the processor, can cause the processor to perform a series of operational steps to generate a computer-implemented process, such that the instructions, when executed on the processor, provide steps for implementing the actions specified in each flowchart block(s). The computer program instructions can also cause at least some of the operational steps illustrated in each flowchart block to be performed in parallel. Furthermore, some of the steps may also be executed across more than one processor, as may occur in a multiprocessor computer system. Furthermore, one or more blocks or combinations of blocks in each flowchart diagram can be executed concurrently with other blocks or combinations of blocks, or in a different order than that illustrated, without departing from the scope or spirit of the present invention.

[0234] Thus, each block of each flowchart diagram supports a combination of means for performing the specified operations, a combination of steps for performing the specified operations, and program instruction means for performing the specified operations. It should also be understood that each block of each flowchart diagram and combinations of blocks in each flowchart diagram can be implemented by a dedicated hardware-based system that performs the specified actions or steps, or a combination of dedicated hardware and computer instructions. The foregoing examples should not be construed as limiting or exhaustive, but rather as illustrative use cases to illustrate at least one implementation of various embodiments of the present invention.

[0235] Additionally, in one or more embodiments (not shown), the logic of the exemplary flowcharts may be executed using an embedded logic hardware device, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable array logic (PAL), or the like, or a combination thereof, instead of a CPU. The embedded logic hardware device may directly execute its embedded logic to perform actions. In one or more embodiments, a microcontroller may be configured to directly execute its own embedded logic to perform actions and access its own internal memory and its own external input and output interfaces (e.g., hardware pins or a wireless transceiver) to perform actions, such as a system on a chip (SOC).

Claims

1. A computer system having one or more processors, Providing a plurality of characters, each encoded as one or more code units based on a character code, wherein the one or more code units for each character represent a code point of a character encoding scheme; determining one or more processor features in the one or more processors, the one or more processor features enabling parallel processing of the one or more code units simultaneously; determining an encoding model based on the character code, the one or more processor functions, and a target character code; and using the one or more processor functions to convert the one or more code units into one or more target code units based on the encoding model, wherein the one or more target code units are based on the target character code, and the one or more target code units encode the code points on a character-by-character basis; providing a plurality of target characters to a target stream, each target character being encoded as said one or more target code units; A method comprising:

2. using the one or more processor facilities to transform the one or more code units into the one or more target code units; determining one or more instructions associated with the one or more processor functions based on the encoding model; 2. The method of claim 1, further comprising: executing the one or more instructions to simultaneously convert one or more portions of the one or more code units into one or more portions of the one or more target code units.

3. said converting said one or more code units into said one or more target code units comprises: loading a first portion of the one or more code units into a first register, each code unit being two bytes wide; loading a second portion of the one or more code units into a second register, each code unit being two bytes wide, the second register being the same width as the first register; 2. The method of claim 1, further comprising: providing the one or more target code units based on packing the first portions of the one or more code units and the second portions of the one or more code units into a third register, wherein each target code unit is one byte wide and the third register is the same width as the register.

4. said converting said one or more code units into said one or more target code units comprises: loading the one or more code units into a register, each code unit being one byte wide, and three or more code units being used to encode each code point representing each character; 2. The method of claim 1, further comprising: generating the one or more target code units in the register based on the encoding model, each target code unit being two bytes wide, and each target code unit encoding the code points representing a respective character.

5. 2. The method of claim 1, wherein the character encoding scheme is Unicode, the character code is one or more of Unicode Transformation Format-8 bit (UTF-8), Unicode Transformation Format-16 bit (UTF-16), or Unicode Transformation Format-32 bit (UTF-32), and the target character code is one or more of UTF-8, UTF-16, or UTF-32.

6. determining the encoding model based on the character code categorizing the one or more code units or the one or more target code units based on the number of bytes used to represent each character, each category using the number of bytes to represent each character; The method of claim 1 , further comprising: further determining the coding model based on the category of the one or more coding units or the one or more target code units.

7. The method of claim 1 , further comprising displaying the one or more encoded target code units in a visualization with respect to one or more of a label, a description, a mark, or metadata.

8. A computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a computer system, cause the computer system to: Providing a plurality of characters, each encoded as one or more code units based on a character code, wherein the one or more code units for each character represent a code point of a character encoding scheme; determining one or more processor features in the one or more processors, the one or more processor features enabling parallel processing of the one or more code units; determining an encoding model based on the character code, the one or more processor features, and a target character code; using the one or more width registers to convert the one or more code units to one or more target code units stored in the one or more processor functions based on the encoding model, the one or more target code units being based on the target character code, and the one or more target code units encoding the code points on a character-by-character basis; providing a plurality of target characters to a target stream, each target character being encoded as the one or more target code units.

9. converting the one or more code units into the one or more target code units using the one or more processor functions; determining one or more instructions associated with the one or more processor functions based on the encoding model; 9. The computer-readable storage medium of claim 8, further comprising: executing the one or more instructions to simultaneously convert one or more portions of the one or more code units to one or more portions of the one or more target code units.

10. said converting said one or more code units into said one or more target code units comprises: loading a first portion of the one or more code units into a first register, each code unit being two bytes wide; loading a second portion of the one or more code units into a second register, each code unit being two bytes wide, the second register being the same width as the first register; 9. The computer-readable storage medium of claim 8, further comprising: providing the one or more target code units based on packing the first portions of the one or more code units and the second portions of the one or more code units into a third register, wherein each target code unit is one byte wide and the third register is the same width as the first register.

11. said converting said one or more code units into said one or more target code units comprises: loading the one or more code units into a register, each code unit being one byte wide, and three or more code units being used to encode each code point representing each character; 9. The computer-readable storage medium of claim 8, further comprising: generating the one or more target code units in the register based on the encoding model, each target code unit being two bytes wide, and each target code unit encoding the code points representing a respective character.

12. 9. The computer-readable storage medium of claim 8, wherein the character encoding scheme is Unicode, the character code is one or more of Unicode Transformation Format-8 bit (UTF-8), Unicode Transformation Format-16 bit (UTF-16), or Unicode Transformation Format-32 bit (UTF-32), and the target character code is one or more of UTF-8, UTF-16, or UTF-32.

13. determining the encoding model based on the character code categorizing the one or more code units or the one or more target code units based on the number of bytes used to represent each character, each category using the number of bytes to represent each character; and further determining the encoding model based on the category of the one or more coding units or the one or more target code units.

14. One or more processors; a memory in which one or more programs are stored; wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising: Providing a plurality of characters, each encoded as one or more code units based on a character code, wherein the one or more code units for each character represent a code point of a character encoding scheme; determining one or more processor features in the one or more processors, the one or more processor features enabling parallel processing of the one or more code units; determining an encoding model based on the character code, the one or more width registers, and a target character code; using the one or more process functions to convert the one or more code units into one or more target code units stored in the one or more process functions based on the encoding model, the one or more target code units being based on the target character code, the one or more target code units encoding the code points for each character; providing a plurality of target characters to a target stream, each target character being encoded as the one or more target code units.

15. converting the one or more code units into the one or more target code units using the one or more processor functions; determining one or more instructions associated with the one or more processor functions based on the encoding model; 15. The computer system of claim 14, further comprising: executing the one or more instructions to simultaneously convert one or more portions of the one or more code units into one or more portions of the one or more target code units.

16. The converting of the one or more code units into the one or more target code units comprises: loading a first portion of the one or more code units into a first register, each code unit being two bytes wide; loading a second portion of the one or more code units into a second register, each code unit being two bytes wide, the second register being the same width as the first register; 15. The computer system of claim 14, further comprising: providing the one or more target code units based on packing the first portions of the one or more code units and the second portions of the one or more code units into a third register, wherein each target code unit is one byte wide and the third register is the same width as the first register.

17. said converting said one or more code units into said one or more target code units comprises: loading the one or more code units into a register, each code unit being one byte wide, and three or more code units being used to encode each code point representing each character; 15. The computer system of claim 14, further comprising: generating the one or more target code units in the register based on the encoding model, each target code unit being two bytes wide, and each target code unit encoding the code points representing a respective character.

18. 15. The computer system of claim 14, wherein the character encoding scheme is Unicode, the character code is one or more of Unicode Transformation Format-8 bit (UTF-8), Unicode Transformation Format-16 bit (UTF-16), or Unicode Transformation Format-32 bit (UTF-32), and the target character code is one or more of UTF-8, UTF-16, or UTF-32.

19. determining the encoding model based on the character code categorizing the one or more code units or the one or more target code units based on the number of bytes used to represent each character, each category using the number of bytes to represent each character; and further determining the encoding model based on the category of the one or more code units or the one or more target code units.

20. 15. The computer system of claim 14, further comprising displaying the one or more encoded target code units in a visualization of one or more of a label, a description, a mark, or metadata.

Images