Navigation aid for visually impaired

Abstract

Methods for navigation assistance for vision-impaired individuals can include determining that an image captured by an imaging device contains an unknown object, and determining that the unknown object is not resolvable within a threshold period of time. The methods can further include performing an operation to reallocate computing resources among memory devices that are couplable to the imaging device in response to determining that the unknown object is not resolvable within the threshold period of time. Data corresponding to the unknown object can be written to the reallocated computing resources, and operations involving the data corresponding to the unknown object can be performed using the reallocated computing resources to resolve the unknown object.

Description

Technical Field

[0001] This disclosure generally relates to semiconductor memories and methods, and more specifically, to devices, systems and methods for navigation assistance for visually impaired persons. Background Technology

[0002] Memory devices are typically provided as internal components of computers or other electronic systems, in semiconductors, or integrated circuits. Many different types of memory exist, including volatile and non-volatile memory. Volatile memory may require power to retain its data (e.g., host data, erroneous data), and includes Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Synchronous Dynamic Random Access Memory (SDRAM), and Thyristor Random Access Memory (TRAM), among others. Non-volatile memory provides permanent data by retaining stored data when no power is supplied, and includes NAND flash memory, NOR flash memory, and resistive variable memory such as Phase-Change Random Access Memory (PCRAM), Resistive Random Access Memory (RRAM), and Magnetoresistive Random Access Memory (MRAM), such as Spin Torque Transfer Random Access Memory (STT RAM), etc.

[0003] A memory device may be coupled to a host computer (e.g., a host computing device) to store data, commands, and / or instructions for use by the host computer or electronic system during operation. For example, data, commands, and / or instructions may be transferred between the host computer and the memory device during operation of the computing or other electronic system. Summary of the Invention

[0004] In one aspect, this disclosure relates to a method for navigation assistance for visually impaired individuals, comprising: determining, by a processor, that an image captured by an imaging device coupled to the processor contains an unknown object, the processor being coupled to a first memory device comprising a first type of media and a second memory device comprising a second type of media; determining, at least in part, based on an object recognition model executed by the processor, that at least a portion of the unknown object represents an unresolvable object to be resolved within a threshold time period; in response to determining that the unknown object is unresolvable within the threshold time period, performing operations to reallocate computing resources between the first memory device and the second memory device; having the processor write at least a portion of data associated with the unknown object to the reallocated resources of the first memory device or the second memory device or both; and using the reallocated computing resources to perform operations involving data corresponding to the unknown object to resolve the unknown object.

[0005] In another aspect, this disclosure relates to a navigation aid for visually impaired individuals, comprising: a first memory device including a first type of media; a second memory device including a second type of media; and a processor coupled to the first and second memory devices, wherein the processor will: determine that an image captured by an imaging device coupled to the processor contains an unknown object; determine, at least in part, that the unknown object is unresolvable within a threshold time period based on a determined confidence level associated with the image; in response to determining that the unknown object is unresolvable within the threshold time period, perform operations to reallocate computing resources between the first and second memory devices; write at least a portion of data associated with the unknown object to the reallocated resources of the first or second memory device or both; and use the reallocated computing resources to perform operations involving data corresponding to the unknown object to resolve the unknown object.

[0006] In another aspect, this disclosure relates to a navigation assistance system for visually impaired individuals, comprising: a mobile computing device including a processor, a first memory device, a second memory device, and a third memory device; and an imaging device residing on the mobile computing device and coupled to the processor, wherein the processor will: receive an image captured by the imaging device, the image containing more than a threshold number of unidentifiable pixels; classify the captured image as an image containing unknown objects based on the presence of more than the threshold number of unidentifiable pixels; and determine at least a portion of the unidentifiable pixels in a certain number of seconds based at least in part on an object recognition model executed by the processor or a determined confidence level associated with the captured image, or both. Unresolvable within a threshold time period; determining that the unknown object poses an impending danger to the user of the mobile computing device; in response to determining that at least a portion of the unrecognizable pixels are unresolvable within the first threshold time period, or determining that the unknown object poses an impending danger to the user of the mobile computing device, or both, reallocating computing resources in the first memory device, the second memory device, or the third memory device, or any combination thereof; using the reallocated computing resources to perform operations involving at least a portion of the unrecognizable pixels corresponding to the unknown object to resolve the unknown object; and determining whether the operation for resolving the unknown object was successful within a second threshold time period. Attached Figure Description

[0007] Figure 1 This is a functional block diagram of a device including a host and a memory device according to various embodiments of the present disclosure.

[0008] Figure 2This is another functional block diagram in the form of a computing system including a device according to several embodiments of the present disclosure, the device including a host and a memory system.

[0009] Figure 3 This is a functional block diagram of a device comprising a memory system according to various embodiments of the present disclosure.

[0010] Figure 4 This is another functional block diagram in the form of a device including a memory system, according to several embodiments of the present disclosure.

[0011] Figure 5 These are diagrams illustrating a mobile computing device and an unknown object according to various embodiments of the present disclosure.

[0012] Figure 6 The flowchart represents an example method for navigation assistance for visually impaired persons, according to several embodiments of the present disclosure. Detailed Implementation

[0013] This document describes methods, devices, and systems related to navigation assistance for visually impaired individuals. As described below, a method for navigation assistance for visually impaired individuals may include determining that an image captured by an imaging device contains an unknown object. Such a method may include determining that the unknown object is unresolvable within a threshold time period. Operations may be performed in response to determining that the unknown object is unresolvable within the threshold time period to reallocate computational resources among memory devices that can be coupled to the imaging device. Data corresponding to the unknown object may be written to the reallocated computational resources, and operations involving the data corresponding to the unknown object may be performed using the reallocated computational resources to resolve the unknown object.

[0014] One of the many challenges faced by visually impaired individuals (e.g., blind, deaf-blind, or partially blind) is navigating a world primarily designed for sighted people. For example, visually impaired individuals may struggle to navigate cities, towns, shopping malls, shops, and other locations to perform daily activities and tasks without relying on their eyesight. To alleviate some of the challenges associated with providing accurate and reliable navigation for visually impaired individuals, various technologies have been developed, typically relying on the Global Positioning System (GPS) for navigation.

[0015] Some of these GPS-based methods are implemented in standalone devices, while others are provided as applications executed by mobile computing devices such as smartphones. As used herein, the term "mobile computing device" generally refers to a handheld computing device (e.g., a smartphone) having a tablet or phablet form factor. Typically, a tablet form factor may include a display between approximately 3 inches and 5.2 inches (diagonally measured), while a phablet form factor may include a display between approximately 5.2 inches and 7 inches (diagonally measured). However, instances of "mobile computing device" are not limited to these, and in some embodiments, "mobile computing device" may refer to IoT devices, as well as other types of edge computing devices.

[0016] While the aforementioned GPS-based methods can alleviate some of the navigation challenges faced by visually impaired individuals, they may exhibit various drawbacks. For example, GPS-based navigation devices and / or applications may not provide accurate routes in enclosed spaces such as shopping malls, subways, or enclosed parking lots, where there may be no direct line of sight between GPS information and the navigation device used by the visually impaired individual. Due to the inability to provide accurate navigation in such enclosed spaces, some methods may provide inaccurate routes that could mislead or confuse visually impaired individuals, thus exacerbating the challenges they already face daily.

[0017] Furthermore, some navigation assistance methods, particularly those designed for visually impaired individuals, may fail to respond to the introduction of unknown (e.g., unidentifiable) objects into the route a visually impaired person is traversing. For example, if an unknown object is introduced into the route a visually impaired person is traversing in the direction of a device and / or application popular in some methods, these devices and / or applications may not be able to resolve such unknown objects accurately and / or in a timely manner. For instance, in cases where the unknown object is captured or otherwise detected by an imaging device associated with a computing device and therefore may not be able to resolve such unknown objects accurately and / or in a timely manner, some methods may not be able to adequately allocate computing resources within the computing device (e.g., a mobile computing device). As used herein, the terms “resolve” and “resolution” generally refer to identifying and / or determining what an unknown object is, depending on the context.

[0018] In situations where the presence of an unknown object poses an impending danger or threat to a visually impaired person, there is a risk of physical injury or harm to that person. For example, if a visually impaired person is navigating outdoors using a navigation aid operating according to the methods described above, and an unknown object is on their path, the limitations of the methods may result in the visually impaired person being struck by the unknown object and potentially injured. As an example, if a visually impaired person is navigating in a park where people are engaged in various recreational activities, a baseball, softball, soccer ball, frisbee, etc. (e.g., an unknown object) may mistakenly strike and step onto a path that could lead to a collision with the visually impaired person. In this example, the methods described above may not be able to detect and / or resolve the unknown object, and / or provide instructions to the visually impaired person to change their route before the unknown object strikes them.

[0019] It should be noted that the examples above are illustrative in nature and should not be construed as limiting the scope of this disclosure. For example, other types of unknown objects are anticipated in addition to those listed above. Several additional non-limiting examples of objects that may appear as unknown objects and pose a danger to visually impaired persons include objects that may be placed in different locations at different times of day or on different dates, such as sandwich panels, traffic cones, temporary road construction signs, etc. In addition, or in alternatives, objects such as bicycles, scooters, trash cans, trash bags, garbage, construction debris, etc., may be present along a visually impaired person's route at specific times and not at other times, thus creating a potentially dangerous situation if the navigation device fails to detect and resolve them. Another non-limiting example of objects that may appear as unknown objects and potentially enter a visually impaired person's route is an animal (domestic or other), which may pose a danger to visually impaired persons if not accurately and / or quickly resolved.

[0020] To address these and other problems associated with some currently implemented methods, aspects of this disclosure may allow for accurate and / or timely resolution of unknown objects, particularly in navigation assistance for visually impaired individuals. For example, the embodiments described herein may allow for timely and accurate resolution of unknown objects by purposefully reallocating computing resources (e.g., processing resources and / or memory resources) available for mobile computing devices, such that the most efficient (e.g., fastest, most accurate, etc.) computing resources are available as needed to process information about the unknown object to resolve it, and allow visually impaired users of mobile computing devices to navigate safely in the presence of unknown objects, especially if the unknown object is determined to pose an impending danger to the visually impaired individual.

[0021] As described in more detail herein, in some embodiments, the mobile computing device may perform operations to resolve an unknown object without transmitting information corresponding to the unknown object to a location outside the mobile computing device. For example, in some embodiments, the mobile computing device may perform operations to reallocate computing resources available to the mobile computing device so that the most efficient computing resources can be used as needed to process information about the unknown object, in order to resolve the unknown object without transmitting information corresponding to the unknown object to a location outside the mobile computing device, such as a base station or other processing circuitry outside the mobile computing device.

[0022] However, in addition to communicating with a GPS network or without communicating with a GPS network, some embodiments described herein allow the mobile computing device to communicate with one or more base stations and / or Wi-Fi hotspots. This allows the mobile computing device to receive more localized and up-to-date route information compared to methods that rely solely on GPS-based navigation. Furthermore, in some embodiments, the mobile computing device can transmit information to and from base stations and / or Wi-Fi hotspots as part of the process of resolving unknown objects. That is, in some embodiments, the mobile computing device can transmit information corresponding to an unknown object (e.g., an image of the unknown object captured by an imaging device associated with the mobile computing device) to a base station, and the base station can assist in identifying the unknown object.

[0023] Furthermore, various aspects of this disclosure seek to improve the performance of computing systems (e.g., mobile computing devices) when processing applications (e.g., navigation assistance applications for visually impaired individuals) that may generate demanding workloads that will be processed by the mobile computing device. As used herein, the term "application" generally refers to one or more computer programs that may contain computational instructions executable to enable the computing system to perform certain tasks, functions, and / or activities. The amount of computing resources (e.g., processing resources and / or memory resources) consumed during the execution of an application can be measured in terms of "workload." As used herein, the term "workload" generally refers to the aggregate computing resources consumed during the execution of an application performing a task, function, and / or activity. During the execution of an application, the computing system 100 may execute multiple sub-applications, subroutines, etc. The amount of computing resources consumed during the execution of an application (including sub-applications, subroutines, etc.) can be referred to as the workload.

[0024] Some applications that generate demanding workloads include those that process data such as images and / or video in real time. Such applications may request significant computing resources and thus generate demanding workloads, especially when requiring real-time processing of high-quality images and / or video to correct defects in the images and / or video. Examples of these types of applications may include those designed to provide navigation assistance to visually impaired users who can rely on real-time captured images and / or video to make decisions that could affect the safety of the visually impaired user of the application.

[0025] As workload demands increase, especially given advancements in broadband cellular network technology, the challenges associated with workload processing optimization are likely to be exacerbated in mobile computing devices (e.g., smartphones, tablets, phablets, and / or Internet of Things (IoT) devices), where physical space limits the amount of processing and / or memory resources available for the device. Furthermore, in some approaches, performing demanding workloads on mobile computing devices can rapidly deplete the battery resources available to the device and / or generate undesirable thermal behavior (e.g., the device may overheat and become unable to operate stably).

[0026] To attempt to perform demanding workloads on mobile computing devices, some methods may involve tuning the device's performance during certain workloads to ensure sufficient computing resources are available. Other methods may involve tuning the device's performance during certain workloads to mitigate the adverse effects on battery consumption and / or thermal behavior. However, such methods may therefore utilize only a subset of available computing resources and / or fail to utilize them at all. This can be particularly problematic in mobile computing devices, as mentioned above, which may already have reduced computing resources due to space constraints compared to, for example, desktop computing devices.

[0027] In contrast, the embodiments described herein provide hardware circuitry (e.g., controllers, processors, etc.) that can monitor and / or determine the characteristics of a workload being executed in a computing system or mobile computing device when the data corresponding to the workload is stored in different types of memory devices. Based on the monitored or determined characteristics of the workload, the hardware circuitry can write at least a portion of the workload to different types of memory devices. For example, if the workload is executed when the data corresponding to the workload is stored in a volatile memory device, and if the data corresponding to the workload is stored in a non-volatile memory device, and the hardware circuitry determines that the execution of the workload can be optimized, then the hardware circuitry can write at least a portion of the data corresponding to the workload to the non-volatile memory device. This dynamic determination of workload characteristics and subsequent allocation of the workload to memory devices containing different types of media can be particularly beneficial in mobile computing systems, especially when an increasing number of processing resource-intensive workloads are being executed on mobile computing devices.

[0028] Non-limiting examples of how to optimize workloads can include optimizing the battery consumption of the computing system, the bandwidth associated with the computing system, the computing resource consumption associated with the computing system, and / or the execution speed of the computing system for the workload, etc. For example, if the computing system is a mobile computing device (e.g., a smartphone, IoT device, etc.), the battery power of the computing device may be consumed rapidly when executing workloads involving certain types of high-power memory devices. Therefore, in order to optimize the battery power consumption of, for example, a mobile computing device, the hardware circuitry can cause at least a portion of the data corresponding to the workload to be written to a memory device characterized by low power consumption when executing the workload.

[0029] Another non-limiting example of workload optimization may include optimizing workload execution by utilizing memory devices and / or media types that exhibit different memory capacity and bandwidth capabilities. For example, memory devices exhibiting high capacity but low bandwidth (e.g., NAND memory devices) may be used for some types of workloads (or portions thereof), while memory devices exhibiting high bandwidth but low capacity (e.g., 3D stacked SDRAM memory devices) may be used for some types of workloads (or portions thereof). By utilizing the capacity of memory devices that exhibit high capacity but low bandwidth or vice versa for different workloads, the embodiments described herein can optimize the amount of time, processing resources, and / or power consumed when executing resource-intensive applications on a computing device or mobile computing device. However, the embodiments are not limited thereto, and other examples of optimizing workload execution according to this disclosure are described in more detail herein.

[0030] As described in more detail herein, embodiments can further optimize workload execution in a mobile computing system by writing workload-related data to a memory device based on data characteristics (e.g., the frequency of data accesses involved in performing the workload). Data access frequency can refer to the amount of data accessed (e.g., reads, writes, etc.) involved in performing the workload. The frequency of data access may be referred to herein with respect to “hot data” and “cold data.” As used herein, “cold data” means that a particular memory object has not been accessed for a long period of time relative to other memory objects read from the memory device. As used herein, “hot data” means that a particular memory object has been accessed frequently relative to other memory objects read from the memory device.

[0031] For example, if certain data involved in the execution of a workload is determined to be "hot," such data can be written to a memory device containing a media type well-suited for fast data access. A non-limiting example of a memory device to which hot data can be written during the execution of the workload described herein is a volatile memory device, such as a DRAM device.

[0032] In contrast, if some data involved in the execution of a workload is determined to be "cold," such data can be written to a memory device containing a media type well-suited for storing data that is not frequently accessed. A non-limiting example of a memory device to which cold data can be written during the execution of the workload described herein is a non-volatile memory device, such as a NAND flash memory device.

[0033] In the following detailed description of this disclosure, reference is made to the accompanying drawings, which form a part of this disclosure, and the drawings illustrate by way of illustration one or more embodiments of this disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice embodiments of this disclosure, and it should be understood that other embodiments may be utilized and that process, electrical, and structural changes may be made without departing from the scope of this disclosure.

[0034] As used herein, designators such as “N”, “M”, etc., specifically relating to reference numerals in the drawings, indicate that a plurality of such specific features may be included. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” may include both singular and plural referents. Additionally, “a plurality,” “at least one,” and “one or more” (e.g., a plurality of memory banks) may refer to one or more memory banks, while “a plurality” is intended to refer to more than one such thing.

[0035] Furthermore, the words “may” and “can” are used throughout this application in a permissive sense (i.e., having the potential to be) rather than in a mandatory sense (i.e., must). The term “comprising” and its derivatives mean “including but not limited to”. Depending on the context, the term “coupled / coupling” means a physical, direct or indirect connection or access to and movement (transmission) of commands and / or data. Depending on the context, the terms “data” and “data value” are used interchangeably herein and may have the same meaning.

[0036] The diagrams in this document follow a numbering rule, where the first one or more digits correspond to the diagram number, and the remaining digits identify elements or components within the diagram. Similar elements or components between different diagrams can be identified by using similar digits. For example, 104 can be referenced. Figure 1 Component "04" in the text, and similar components in Figure 2 The symbol 204 may be used in this document. A single element symbol is typically used to refer to a group or more similar elements or components. For example, multiple reference elements, such as elements 543-1 to 543-N (or, in alternatives, 543-1, ..., 543-N), may generally be referred to as 543. As will be understood, elements shown in the various embodiments herein may be added, interchanged, and / or removed to provide multiple additional embodiments of this disclosure. Furthermore, the scale and / or relative dimensions of the elements provided in the figures are intended to illustrate certain embodiments of this disclosure and should not be construed as limiting.

[0037] Figure 1 This is a functional block diagram of a computing system 100 comprising a device according to various embodiments of the present disclosure, the device including a host 102 and a memory system 104. As used herein, "device" may refer to, but is not limited to, any of a variety of structures or combinations thereof, such as a circuit or circuit system, one or more dies, one or more modules, one or more devices, or one or more systems. In some embodiments, the computing system 100 may be a mobile computing system (e.g., a mobile computing device, such as...). Figure 5 The mobile computing device 501 described herein may be a smartphone, tablet computer, phablet, and / or IoT device, etc. The memory system 104 may include multiple different memory devices 123, 125 (and / or as described herein). Figure 2 As described in section 227), it may include one or more different media types 123, 125 (and / or as used herein). Figure 2 (as described in 227). Different memory devices 123, 125 and / or 227 may include one or more memory modules (e.g., single in-line memory modules, dual in-line memory modules, etc.).

[0038] Memory system 104 may include volatile memory and / or non-volatile memory. In several embodiments, memory system 104 may include a multi-chip device. The multi-chip device may include multiple different memory devices 123, 125, and / or 227, which may include multiple different memory types and / or memory modules. For example, the memory system may include non-volatile or volatile memory on any type of module. Figure 1 As shown, computing system 100 may include controller 120, which may include processor 122. Each of the components (e.g., host 102, controller 120, processor 122 and / or memory devices 123, 125) may be individually referred to herein as a “device”.

[0039] Memory system 104 may provide main memory for computing system 100, or may be used as additional memory and / or storage devices throughout computing system 100. Memory system 104 may include one or more memory devices 123, 125, which may contain volatile and / or non-volatile memory cells. For example, at least one of memory devices 123, 125 may be a flash array with a NAND architecture. Furthermore, at least one of memory devices 123, 125 may be a dynamic random access array of memory cells. Embodiments are not limited to a specific type of memory device. For example, memory system 104 may include RAM, ROM, DRAM, SDRAM, PCRAM, RRAM, and / or flash memory (e.g., NAND and / or NOR flash memory devices), etc.

[0040] In some embodiments, memory system 104 may include other non-volatile memory devices 123, 125, such as non-volatile random access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), "emerging" memory devices such as variable resistance (e.g., 3-D crosspoint (3D XP)) memory devices, memory devices including self-select memory (SSM) cell arrays, etc., or any combination thereof.

[0041] Variable resistance memory devices can combine stackable cross-grid data access arrays to perform bit storage based on changes in bulk resistance. Furthermore, compared to many flash-based memories, variable resistance non-volatile memories can perform in-situ write operations, where non-volatile memory cells can be programmed without pre-erasing them. Compared to flash-based memories and variable resistance memories, selectable memory cells can comprise memory cells made of a single chalcogenide material that serves as both a switch and a storage element for the memory cell.

[0042] like Figure 1As shown, memory devices 123 and 125 can be different types of memory devices. For example, memory device 125 can be a persistent non-volatile memory device, such as a 3D XP memory device or a NAND memory device, etc., and memory device 123 can be a non-persistent volatile memory device, such as a DRAM device, or vice versa. Therefore, memory devices 123 and 125 can include different media types 124 and 126. However, the embodiments are not limited thereto, and memory devices 123 and 125 can include any type of memory device, provided that at least two of memory devices 123 and 125 include different media types 124 and 126. As used herein, "media type" generally refers to the type of memory cell architecture corresponding to memory devices 123 and 125. For example, one of media types 124 and 126 can correspond to a memory cell array including at least one capacitor and at least one transistor, while the other of media types 124 and 126 can include an array of floating gate metal-oxide-semiconductor field-effect transistors. In some embodiments, at least one of media types 124, 126 may include an array of variable resistance memory cells configured to perform bit storage based on changes in the volume resistance associated with the variable resistance memory cells.

[0043] like Figure 1 As illustrated, host 102 may be coupled to memory system 104. In several embodiments, memory system 104 may be coupled to host 102 via one or more channels (e.g., channel 103). Figure 1 In this configuration, memory system 104 is coupled to host 102 via channel 103, which may also be coupled to controller 120 and / or processor 122 of memory system 104. Controller 120 and / or processor 122 are coupled to memory devices 123 and 125 via channels 105 and 107. In some embodiments, each of memory devices 123 and 125 is coupled to controller 120 and / or processor 122 by one or more corresponding channels 105 and 107, such that each of memory devices 123 and 125 can receive messages, commands, requests, protocols, or other signaling conforming to the type of memory device 123 or 125 coupled to controller 120 (e.g., messages, commands, requests, protocols, or other signaling conforming to media types 124 and 126 of memory devices 123 and 125).

[0044] The computing system 100 may further include an imaging device 121. The imaging device 121 may be communicatively coupled to the host 102 and / or the memory system 104 (e.g., controller 120 and / or processor 122). The imaging device 121 may be a camera, an ultrasonic scanner, an ultrasound device, a stereo imaging device, a magnetic resonance imaging device, an infrared imaging device, or other imaging devices capable of capturing data containing images or image streams (e.g., streaming video and / or “live video”) in real time and transmitting information corresponding to the images and / or image streams to the computing system 100. Generally, the imaging device may be any mechanical, digital, or electronic viewing device; a still camera; a video camera; a cinema camera; or any other instrument, device, or format capable of recording, storing, or transmitting images, video, and / or information.

[0045] As used herein, the term "live video" and variations thereof generally refer to a sequence of images that are captured and processed, reproduced, and / or broadcast simultaneously (or nearly simultaneously). In some embodiments, alternatively, "live" video may be referred to herein as "data captured by an imaging device" or "data captured from an imaging device." Furthermore, as used herein, the term "streaming video" and variations thereof generally refer to a sequence of images captured by an imaging device and subsequently processed, reproduced, and / or broadcast. In some embodiments, alternatively, "streaming" video may be referred herein as "data captured by an imaging device" or "data captured from an imaging device."

[0046] Typically, such data captured by the imaging device (e.g., images, image streams, and / or “live” video) may be displayed or broadcast on a viewing device (e.g., the display screen of a mobile computing device) associated with the computing system 100, and / or processed by a processor (e.g., processor 122) for a certain threshold time period following capture by the imaging device. In some embodiments, the data captured by the imaging device may be displayed, broadcast, and / or processed for a threshold time period relative to the capture by the imaging device, which is approximately seconds or minutes compared to hours or days. This data (e.g., images and / or video streams) may include any live or recorded media content transmitted to or from the computing system 100 (e.g., a mobile computing device) via a connection path, such as a wired communication channel, and / or a wireless communication channel, such as the Internet. Therefore, as described in more detail herein, data may be captured by imaging device 121 and subsequently stored in memory devices 123, 125 coupled to imaging device 121, processed by processor 122 associated with memory devices 123, 125, and subsequently broadcast, and / or data may be captured by imaging device 121, stored in memory devices 123, 125 coupled to imaging device 121, processed by processor 122 associated with memory devices 123, 125, and / or broadcast in real time (or near real time based on latency during transmission between the various components described herein) when data is captured by imaging device 121.

[0047] In some embodiments, imaging device 121 may capture data containing images used in the execution of navigation assistance operations, such as images and / or streaming video (e.g., live video). As used herein, “navigation assistance operation” generally refers to one or more operations performed and / or executed by a hardware circuitry system (e.g., processor 122, imaging device 121, and / or at least one of memory devices 123, 125) to provide route and / or other navigation information to a user of computing system 100 (e.g., a visually impaired user). In some embodiments, navigation assistance operations may be performed by executing an application running on computing system 100.

[0048] In some embodiments, the images and / or streaming video captured by imaging device 121 may include images and / or streaming video of various objects within or near a route generated as part of performing navigation assistance operations, some of which may not be immediately recognizable by computing system 100 and thus classified as “unknown objects.” As used herein, the term “not immediately recognizable” and variations thereof generally refer to a situation where an object captured by imaging device 121 cannot be resolved within a threshold time period. In some embodiments, whether an object can be resolved within the threshold time period may be determined based on a confidence level associated with one or more pixels of the image and / or streaming video. As used herein, the term “confidence level” generally refers to a calculated probability that the corresponding pixel of the image and / or streaming video displays a value sufficiently accurate for the image to be recognized. If a number of pixels greater than a threshold amount of the image and / or streaming video displays below a certain confidence level, the image and / or streaming video may be classified as “unknown” or “unknown object.” Alternatively, or in some embodiments, whether an object can be resolved within the threshold time period may be determined based on the execution of an object recognition model. As used herein, the term "object recognition model" generally refers to instructions executed by a computing system to identify one or more objects in digital images and / or streaming video. The execution of an object recognition model may include the execution of instructions for detecting, classifying, and / or locating images and / or streaming video in order to parse one or more objects captured in the images and / or streaming video.

[0049] Additionally, it can be determined whether the images and / or streaming video may pose an impending danger to the user of computing system 100. For example, it can be determined that an unknown object is traveling at a speed greater than a threshold, and / or that the unknown object appears poised to come into contact with the user of computing system 100 without any action to redirect the user of computing system 100 away from the path of the unknown object.

[0050] Images and / or streaming video can be captured by imaging device 121 and processed locally within memory system 104 as part of navigation assistance operations (e.g., to provide navigation assistance to visually impaired individuals). By utilizing such aspects of this disclosure, navigation assistance can be performed without transmitting images and / or streaming video to circuitry outside memory system 104. That is, in some embodiments, the images and / or streaming video captured by imaging device 121 can be parsed using circuitry associated with memory system 104 without, for example, hindering host 102. By processing images and / or streaming video within memory system 104, unknown objects can be parsed in a shorter timeframe compared to some other methods, thereby increasing the likelihood of redirecting the user of computing system 100 away from the unknown object if it is determined that the unknown object presents an impending danger to the user of computing system 100.

[0051] Traditionally, the capture and processing / analysis of such images and / or streaming video are computationally intensive processes. For example, applications and the corresponding workloads for processing images and / or streaming video containing unknown objects can be extremely computationally intensive. One reason is that the level of detail captured in such images and / or streaming video can be very detailed and / or can be received quickly and continuously (e.g., in real time), thus making them memory intensive (e.g., due to the detail captured in such images and / or video, the file size corresponding to the images and / or video can be relatively large compared to, for example, a simple photograph).

[0052] However, the embodiments described herein may allow selective processing of workloads relating to images and / or videos corresponding to those captured by imaging device 121, such that workloads corresponding to the execution of applications relating to said images and / or videos are allocated to memory devices 123, 125, 227 to optimize the performance of memory system 104, enabling the resolution of unknown objects as described herein to be implemented using mobile computing devices (e.g., smartphones) among other mobile computing devices described herein.

[0053] Host 102 may be a host system, such as a personal laptop computer, desktop computer, digital camera, smartphone, memory card reader, and / or Internet of Things (IoT)-enabled device, as well as various other types of host. However, in some embodiments, host 102 is a mobile computing device, such as a digital camera, smartphone, memory card reader, and / or Internet of Things (IoT)-enabled device, as well as various other types of host (e.g., in some embodiments, host 102 is not a personal laptop computer or desktop computer). Host 102 may include a system motherboard and / or backplane and may include memory access devices, such as a processor (or processing device).

[0054] Those skilled in the art will understand that "processor" can mean one or more processors, such as in a parallel processing system, multiple coprocessors, etc. Computing system 100 may include a single integrated circuit, or one or more of host 102, memory system 104, control circuitry system 120, and / or memory devices 126-1 to 126-N may be on the same integrated circuit. Computing system 100 may, for example, be a server system and / or a high-performance computing (HPC) system and / or a portion thereof. Although Figure 1 The examples shown illustrate a system with a von Neumann architecture, but embodiments of this disclosure can be implemented in a non-von Neumann architecture that may not include one or more components typically associated with a von Neumann architecture (e.g., CPU, ALU, etc.).

[0055] Memory system 104 may include controller 120, which may include processor 122. Processor 122 may be provided in the form of an integrated circuit, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a reduced instruction set computing device (RISC), an advanced RISC machine, a system-on-a-chip, or other combinations of hardware and / or circuitry configured to perform the operations described in more detail herein. In some embodiments, processor 122 may include one or more processors (e.g., processing devices, coprocessors, etc.).

[0056] Processor 122 may perform operations to monitor and / or determine characteristics of workloads running on memory system 104. These characteristics may include information such as bandwidth consumption, memory resource consumption, access frequency (e.g., whether the data is hot or cold), and / or power consumption while performing the workload. Processor 122 may control the writing of at least a portion of data to different memory devices 123, 125 to optimize workload execution (e.g., optimize operations for parsing unknown objects in navigation assistance applications for visually impaired users), balance workloads across different memory devices 123, 125 for media management purposes, and / or optimize battery consumption of computing system 100, etc.

[0057] In a non-limiting example, a device (e.g., computing system 100) may include a first memory device 123 comprising a first type of media 124 and a second memory device 125 comprising a second type of media 126. In some embodiments, the first type of media 124 and the second type of media 126 each include a set of memory cells exhibiting different storage characteristics. In some embodiments, the first memory device 123, the second memory device 125, and the processor 122 may reside in a mobile computing device, such as a smartphone (e.g., as described herein). Figure 5 The mobile computing device 501 described herein. Processor 122 may be coupled to a first memory device 123 and a second memory device 125. Processor 122 may receive information captured by imaging device 121 which may be coupled to processor 122.

[0058] As used herein, the term "residual" refers to something physically residing on a particular component. For example, residing on a smartphone (e.g., in this context...) Figure 5 The first memory device 123, the second memory device 125, and / or the processor 122 on the computing system 100 and / or mobile computing device 501 described herein refer to the first memory device 123, the second memory device 125, and / or the processor 122 being physically coupled to a smartphone (e.g., as described herein). Figure 5This refers to the case where the computing system 100 and / or mobile computing device 501 described herein are physically located within a smartphone. The term "resident" may be used interchangeably herein with other terms such as "deployed in" or "located in".

[0059] Processor 122 can perform operations to determine that an image captured by imaging device 121 coupled to processor 122 contains an unknown object (e.g., in this document). Figure 5 (Unknown object 547 as described in the image). Processor 122 may determine, at least in part, that an unknown object is unresolvable within a threshold time period based on a determined confidence level associated with the image. In some embodiments, processor 122 may determine, at least in part, that an unknown object is unresolvable within a threshold time period based on an object recognition model executed by the processor.

[0060] In response to determining that an unknown object is unresolvable within a threshold time period, processor 122 may perform operations to reallocate computing resources between a first memory device 123 and a second memory device 125. Continuing this example, processor 122 may write at least a portion of the data associated with the unknown object to the reallocated resources of the first memory device 123 and / or the second memory device 125, and use the reallocated computing resources to perform operations involving the data corresponding to the unknown object to resolve the unknown object. In some embodiments, processor 122 may execute instructions corresponding to one or more machine learning operations as part of performing operations to resolve the unknown object.

[0061] As described above, the first or second memory device may be a non-persistent (e.g., volatile) memory device, and the other of the first or second memory device may be a persistent (e.g., non-volatile) memory device. For example, in some embodiments, the first or second memory device may be a NAND flash memory device comprising a set of single-level memory cells (SLC) and a set of multi-level memory cells (MLC), while the other of the first or second memory device may be a DRAM memory device. In such embodiments, as part of the operation of parsing an unknown object, the processor 122 may write data corresponding to the unknown object to the set of SLC memory cells and / or the DRAM memory device.

[0062] As described above, the processor 122, the first memory device 123, and the second memory device 125 may reside in a mobile computing device (e.g., as described herein). Figure 5 On the mobile computing device 501 described herein. In some embodiments, the processor 122 may determine that the mobile computing device has traveled from the first base station to the second base station (e.g., as described herein). Figure 5The processor 122 handles the handover of the first base station 543-1 and the second base station 543-N as described herein, and performs operations to reallocate computing resources between the first memory device 123 and the second memory device 125 in response to determining that the mobile computing device has undergone a handover. For example, the processor 122 may reallocate or pre-allocate computing resources between the first memory device 123 and the second memory device 125 based on the fact that the mobile computing device has moved from an area covered by a base station network to an area covered by a different base station network.

[0063] As used herein, particularly in the context of network coverage from a base station, the term "network coverage" generally refers to a geographical area characterized by the presence of electromagnetic radiation (e.g., waves with a specific frequency range associated therewith) generated by a base station. As used herein, "base station" generally refers to a device that generates and receives electromagnetic radiation within a specific frequency range and facilitates the transmission of data or other information between the base station and computing devices (e.g., mobile computing devices such as smartphones) within the base station's network coverage area. Several non-limiting examples of the frequency ranges that a base station can generate and receive may include 700MHz to 2500MHz (in the case of a 4G base station) or 28GHz to 39GHz (in the case of a 5G base station).

[0064] In some embodiments, processor 122 may determine that an unknown object presents an impending danger to a user of the mobile computing device, and / or determine that the unknown object cannot be resolved within a threshold time period corresponding to the user's contact with the unknown object. In response to this determination, processor 122 may update route information associated with the mobile computing device to guide the user of the mobile computing device away from the unknown object.

[0065] In some embodiments, when writing a workload to a first memory device 123 or a second memory device 125, the processor 122 may determine the characteristics of the workload. As described herein, the characteristics of the workload may include at least one of the frequency of data access associated with the workload, the latency associated with the execution of the workload, and / or the amount of processing resources consumed when executing the workload. In some embodiments, the application and / or workload may involve processing data received and / or captured by the imaging device 121, for example as part of an operation to parse an unknown object determined to pose an impending danger to a user of a mobile computing device in which the processor 122 is housed. By determining the characteristics of such a workload when executed by different memory devices, the processor 122 may determine which memory devices to write data corresponding to the unknown object to in order to perform operations to parse the unknown object quickly and accurately.

[0066] Figure 1Embodiments may include additional circuitry not described to avoid obscuring the embodiments of this disclosure. For example, memory system 104 may include address circuitry that latches address signals provided on I / O connections via I / O circuitry. Address signals can be received and decoded by row decoders and column decoders to access memory system 104 and / or memory devices 123, 125. Those skilled in the art will understand that the number of address input connections may depend on the density and architecture of memory system 104 and / or memory devices 123, 125.

[0067] Figure 2 This is another functional block diagram of a computing system 200 comprising a device according to various embodiments of the present disclosure, the device including a host 202 and a memory system 204. In some embodiments, the computing system 200 may be a mobile computing system (e.g., a mobile computing device 501, such as a smartphone, tablet computer, phablet, and / or IoT device, etc.). The memory system 204 may include a plurality of different memory devices 223, 225, 227, which may include one or more different media types 223, 225, 227. The different memory devices 223, 225, and / or 227 may include one or more memory modules (e.g., single in-line memory modules, dual in-line memory modules, etc.). The host 202, memory system 204, controller 220, processor 222, memory devices 223, 225, and / or media types 224, 226 may be similar to those described herein. Figure 1 The host 102, memory system 104, controller 120, processor 122, memory devices 123, 125 and / or media types 124, 126 are described in the document.

[0068] In some embodiments, each of memory devices 223, 225, and 227 may be a different type of memory device. Therefore, in some embodiments, each of memory devices 223, 225, and 227 may include different media types 224, 226, and 228. In a non-limiting example, memory device 223 may be a volatile memory device, such as a DRAM device, and may include media type 224 corresponding to a DRAM memory device (e.g., an array of memory cells including at least one capacitor and at least one transistor). Continuing this example, memory device 225 may be a flash memory device, such as a NAND memory device, and may include media type 226 corresponding to a NAND memory device (e.g., an array including floating gate metal-oxide-semiconductor field-effect transistors). In this non-limiting example, memory device 227 may be an emerging memory device (e.g., as described herein). Figure 4The emerging memory device 439 described herein, such as the emerging memory device described above, may include a media type 228 corresponding to the emerging memory device (e.g., an array of resistive variable memory cells configured to perform bit storage based on changes in the volume resistance associated with the resistive variable memory cells).

[0069] Memory devices 223, 225, and 227 may be configured to read, write, and / or store data corresponding to one or more workloads executed by computing system 200. Applications corresponding to the workloads may be executed, for example, by processor 222, such that data written to memory devices 223, 225, and 227 is used to execute applications and / or workloads (e.g., applications and / or workloads associated with providing navigation assistance for visually impaired individuals). As described above, controller 220 may control the writing of at least a portion of data to a memory device different from the memory devices, wherein data is initially written based on characteristics of the workload.

[0070] For example, if data corresponding to a workload for a navigation assistance application is stored in memory device 223, then in response to determining that a different memory device can be used to perform (e.g., optimize) the workload more efficiently, the controller 220 and / or processor 222 may cause at least a portion of the data corresponding to a particular workload to be written to memory devices 225 and / or 227. In some embodiments, causing data corresponding to a particular workload to be written to memory devices 223, 225, and / or 227 may include writing data corresponding to an unknown object to at least one of memory devices 223, 225, and / or 227 to optimize the processing of the data corresponding to the unknown object, thereby resolving the unknown object in the most efficient manner.

[0071] In a non-limiting instance, the system (e.g., in this document) Figure 5 The computing system 200 and / or mobile computing device 501 described herein may include a memory system 204, which includes a processor 222, a first memory device 223 including a first type of media 224, a second memory device 225 including a second type of media 226, and a third memory device 227 including a third type of media 228. In some embodiments, the first memory device 223 may be a dynamic random access memory device, the second memory device 225 may be a NAND flash memory device, and the third memory device 227 may be an emerging memory device, such as a 3D XP memory device, a custom cell memory device, etc., as described above.

[0072] In at least one embodiment, media type 224 includes an array of memory cells comprising at least one capacitor and at least one transistor, media type 226 includes an array of floating-gate metal-oxide-semiconductor field-effect transistors, and media type 228 includes an array of resistive memory cells configured to perform bit storage based on changes in the bulk resistance associated with the resistive memory cells.

[0073] Imaging devices (e.g., those described herein) Figure 1 The imaging device 121 described herein may be coupled to the memory system 204. In this example, the processor 222 may receive an image captured by the imaging device coupled to the processor 222, the image containing a greater than a threshold number of unidentifiable pixels. The processor 222 may classify the captured image as an image containing an unknown object based on the presence of a greater than a threshold number of unidentifiable pixels. In some embodiments, the processor 222 may determine, at least in part, a certain number of unidentifiable pixels that are unresolvable within a first threshold time period based on an object recognition model executed by the processor and / or a determined confidence level associated with the captured image, as described herein.

[0074] Continuing this example, processor 222 can determine whether an unknown object presents an impending danger to the user of the mobile computing device. As described above, whether an unknown object presents an impending danger to the user of the mobile computing device can be based on the trajectory and / or speed of the unknown object and other factors. In response to determining that a certain number of unidentifiable pixels are unresolvable within a first threshold time period and / or determining that an unknown object presents an impending danger to the user of the mobile computing device, processor 222 can reallocate computing resources among a first memory device, a second memory device, or a third memory device, or any combination thereof. In some embodiments, the first time period may include a period during which the unknown object must be identified to allow the user of the mobile computing device to navigate safely away from any impending danger presented by the unknown object.

[0075] In some embodiments, processor 222 may use reallocated computing resources to perform operations involving unidentifiable pixels corresponding to an unknown object to resolve the unknown object, and determine within a second threshold time period whether the operation for resolving the unknown object was successful. If the unknown object is resolved, processor 222 may notify the user of the mobile computing device that the unknown object was resolved in response to the determination that the operation for resolving the unknown object was successful within the second threshold time period. In some embodiments, the second threshold time period may include a period of time during which reallocated computing resources are allowed to be used to resolve the unknown object, allowing the user of the mobile computing device to navigate safely away from any impending danger presented by the unknown object.

[0076] However, if the unknown object is not resolved within the second threshold time period, the processor 222 may update the route information associated with the mobile computing device in response to determining that the operation for resolving the unknown object was unsuccessful within the second threshold time period, so as to guide the user of the mobile computing device away from the unknown object.

[0077] As mentioned above, in some embodiments, at least one of the first memory device 223, the second memory device 225, or the third memory device 227 includes a flash memory device, at least one of the first memory device 223, the second memory device 225, or the third memory device 227 includes a dynamic random access memory device, and at least one of the first memory device 223, the second memory device 225, or the third memory device 227 includes a variable resistance memory device.

[0078] In some embodiments, processor 222 may determine that a first memory device 223, a second memory device 225, or a third memory device 227 exhibits higher bandwidth than another of the first memory device 223, the second memory device 225, or the third memory device 227. In such embodiments, processor 222 may perform operations to reallocate computing resources among the first memory device 223, the second memory device 225, and / or the third memory device 227, such that a greater than a threshold amount of memory exhibiting higher bandwidth is available to perform operations for parsing unknown objects, or a greater than a threshold amount of memory exhibiting faster memory access times is available to perform operations for parsing unknown objects. Processor 222 may then use the reallocated computing resources to perform operations involving unidentifiable pixels corresponding to unknown objects, to use the first memory device 223, the second memory device 225, and / or the third memory device 227 exhibiting higher bandwidth for parsing unknown objects and / or faster memory access times for parsing unknown objects to parse unknown objects.

[0079] As in this article Figure 5 In more detail, processor 222 can determine that the mobile computing device has been transmitted from the first base station (e.g., as described herein). Figure 5 The base station 543-1 described in the document is connected to the second base station (e.g., in this document). Figure 5The handover of base stations 543-N described herein. As used herein, the term "handover" generally refers to the transfer of network coverage from one base station to another. For example, when a mobile computing device moves from one location to another, the mobile computing device may leave the network coverage area provided by the first base station and may enter the network coverage area provided by the second base station. When the mobile computing device begins to receive network coverage from the second base station instead of the first base station, it can be said that a handover has occurred. In response to determining that the mobile computing device has undergone a handover between the first base station and the second base station, the processor 222 may perform operations to reallocate computing resources between the first memory device and the second memory device.

[0080] As described above, in some embodiments, processor 222 may perform operations involving unidentifiable pixels corresponding to an unknown object to resolve the unknown object, as part of executing instructions for a visually impaired user of the assistive mobile computing device. Therefore, in some embodiments, processor 222 may generate vibration patterns corresponding to navigation in the presence of an unknown object, and / or voice-assisted navigation updates corresponding to navigation in the presence of an unknown object, as described herein. Figure 6 As described.

[0081] In some embodiments, at least a portion of the data written to memory device 223, memory device 225, or memory device 227 (e.g., data corresponding to an unknown object) is formatted according to a general or hypothetical digital format. Compared to IEEE 754 floating-point or fixed-point binary formats that include subsets of sign bits, mantissa bits, and exponent bits, a hypothetical general digital format includes subsets of sign bits, state bits, mantissa bits, and exponent bits. This allows hypothetical accuracy, precision, and / or dynamic range to be greater than that of floating-point numbers or other digital formats. Additionally, hypothetical numbers can reduce or eliminate overflow, underflow, NaN, and / or other extreme cases associated with floating-point numbers and other digital formats. Furthermore, the use of delimiters allows fewer bits to be used to represent numerical values ​​(e.g., numbers) compared to floating-point numbers or other digital formats. Therefore, the embodiments described herein allow data corresponding to unknown objects to be converted into a hypothetical format to facilitate efficient processing and parsing of unknown objects.

[0082] As used herein, “precision” refers to the number of bits in a bit string used to perform a computation. For example, if every bit in a 16-bit bit string is used when performing a computation, the bit string may be said to have 16-bit precision. However, if only 8 bits of a 16-bit bit string are used when performing a computation (e.g., if the first 8 bits of the bit string are zero), the bit string may be said to have 8-bit precision. As the precision of the bit string increases, computations can be performed with higher accuracy. Conversely, as the precision of the bit string decreases, computations can be performed with lower accuracy. For example, an 8-bit bit string may correspond to a data range consisting of 255 (256) precision steps, while a 16-bit bit string may correspond to a data range consisting of 65,536 (65,536) precision steps.

[0083] As used herein, "dynamic range" or "dynamic range of data" refers to the ratio between the maximum and minimum values ​​that can be represented by a bit string with a specific precision associated with it. For example, the maximum value that can be represented by a bit string with a specific precision associated with it determines the dynamic range of the bit string's data format. For a bit string in a general number (e.g., assumed) format, the dynamic range can be determined by the values ​​of a subset of the exponent bits of the bit string.

[0084] Dynamic range and / or accuracy may have associated variable range thresholds. For example, the dynamic range of data may correspond to the applications that use the data and / or the various calculations that use the data. This may be due to the fact that one application may expect a different dynamic range than another application, and / or because some calculations may require different data dynamic ranges. Therefore, the embodiments herein may allow the dynamic range of data to be varied to meet the requirements of different applications and / or calculations, such as the resolution of unknown objects captured by an imaging device in the context of providing safe navigation for visually impaired individuals. In contrast to methods that do not allow manipulation of the dynamic range of data to suit the requirements of different applications and / or calculations, the embodiments herein can improve resource utilization and / or data accuracy by allowing the dynamic range of data to be varied based on the applications and / or calculations to which the data is used.

[0085] Figure 3 This is a functional block diagram of a device including a memory system 304 according to several embodiments of the present disclosure. Figure 3 The memory system 304 is described below, and it may be similar to the memory system described in this paper. Figure 1 The memory system 104 and / or the memory system described herein Figure 2 The memory system 204 is described in the document. For example... Figure 3 As shown, the memory system 304 includes a controller 320 (which may be similar to the controller described herein). Figure 1 The controller 120 and / or as described hereinFigure 2 The controller 220 described herein), and the DRAM memory device 331 (which may be similar to the one described herein) Figure 1 One of the memory devices 123 and 125 described herein and / or the memory devices described herein. Figure 2 The memory device 223, 225, 227 described herein, and the NAND memory device 333 (which may be similar to the memory device described herein). Figure 1 One of the memory devices 123 and 125 described herein and / or the memory devices described herein. Figure 2 (One of the memory devices 223, 225, and 227 described herein).

[0086] like Figure 3 As shown, the NAND memory device 333 may include various portions of memory cells, including a set of single-level memory cells (SLC) 335 and a set of three-level memory cells (TLC) 337. Although shown as including SLC 335 and TLC 337, the embodiments are not limited thereto, and the NAND memory device 333 may include one or more sets of multi-level cells (MLC), one or more sets of quad-level cells (QLC), etc. As described above, in some embodiments, the controller 320 may write at least a portion of the data corresponding to an unknown object processed by the memory system 304 as part of executing a navigation assistance application to the SLC portion 335 and / or the TLC portion 337 in order to resolve the unknown object as accurately and quickly as possible.

[0087] In some embodiments, as part of optimizing the performance of memory system 304 during the execution of a workload corresponding to a navigation assistance application, data classified as hot data may be written to SLC portion 335, while data classified as cold data may be written to TLC portion 337, or vice versa. By selectively writing portions of data involved in the workload to different memory portions of NAND memory device 333 (e.g., SLC portion 335 and / or TLC portion 337), the performance of the computing system, particularly during the execution of a workload corresponding to the navigation assistance application described herein, can be improved compared to some other methods. However, embodiments are not limited thereto, and in some embodiments, hot data may be written to DRAM memory device, colder data may be written to NAND memory device 333, and cold data may be written to emerging memory devices (e.g., as described herein). Figure 4 The emerging memory device 439 described in the text.

[0088] For example, by selectively writing data portions corresponding to workloads that benefit from fast execution (e.g., operations for resolving unknown objects presenting an impending danger to a visually impaired person) to DRAM memory device 331, while writing data portions corresponding to workloads that may not significantly benefit from fast execution to SLC portions 335 and / or TLC portions 337, and / or emerging memory devices (e.g., those described herein in...). Figure 4 The emerging memory device 439 described herein can distribute workloads across memory devices within memory system 304, which allows for optimized execution of the workload within memory system 304. For similar reasons, portions of the workload can be written to emerging memory devices (e.g., as described herein). Figure 4 The emerging memory device 439 described in the text.

[0089] Figure 4 This is another functional block diagram in the form of a device including a memory system 404, according to several embodiments of the present disclosure. Figure 4 The memory system 404 is described below, and it can be similar to the memory system described in this paper. Figure 1 The memory system 104 described herein, in Figure 2 The memory system 204 and / or described herein Figure 3 The memory system 304 is described in the document.

[0090] like Figure 4 As shown, the memory system 404 includes a controller 420 (which may be similar to the controller described herein). Figure 1 The controller 120 described in the document, in Figure 2 The controller 220 and / or as described herein Figure 3 The controller 320 described herein), and the DRAM memory device 431 (which may be similar to the one described herein) Figure 1 One of the memory devices 123 and 125 described herein, in Figure 2 One of the memory devices 223, 225, 227 and / or described herein Figure 3 One of the DRAM memory devices 331 described herein), and NAND memory device 433 (which may be similar to the one described herein). Figure 1 One of the memory devices 123 and 125 described herein, in Figure 2 One of the memory devices 223, 225, and 227 described herein, and / or... Figure 3 The NAND memory device 333 described herein, and the emerging memory device 439 (which may be similar to the NAND memory device described herein) Figure 1 One of the memory devices 123 and 125 described herein and / or the memory devices described herein. Figure 2 (One of the memory devices 223, 225, and 227 described herein).

[0091] DRAM memory device 431 may include an array of memory cells, the array including at least one transistor and a capacitor configured to store charge corresponding to a single data bit. NAND memory device 433 may include various portions of memory cells, including a set of single-level memory cells (SLC) 435 and a set of multi-level memory cells (MLC), such as a set of three-level memory cells (TLC) 437, which may be similar to those described herein. Figure 3 The descriptions and explanations are provided in sections 335 (SLC) and 337 (TLC).

[0092] The emerging memory device 439 may be an emerging memory device as described above. For example, the emerging memory device 439 may be a variable resistance (e.g., 3-D crosspoint (3D XP)) memory device, a memory device containing an array of selectable memory (SSM) cells, or any combination thereof.

[0093] As described above, by selectively writing data portions corresponding to workloads that benefit from fast execution (e.g., operations performed by an application to resolve unknown objects that present an impending danger to a visually impaired person) to DRAM memory device 431, while writing data portions corresponding to workloads that may not significantly benefit from fast execution to SLC portion 435 and / or TLC portion 437, and / or emerging memory device 439 (e.g., workloads corresponding to applications that may not be used to identify and resolve objects that present an impending danger to a visually impaired person), workloads can be distributed across memory devices within memory system 404. This allows for optimized execution of workloads within memory system 404. For similar reasons, portions of any workloads that hinder memory system 404 can be written to emerging memory device 439 to facilitate execution of the embodiments described herein.

[0094] Figure 5 These are diagrams illustrating a mobile computing device 501 and an unknown object 547 according to various embodiments of the present disclosure. Figure 5 As shown, the mobile computing device 501 includes an imaging device 521, which can be similar to the one described herein. Figure 1 The imaging device 121 described herein; and the memory system 504, which may be similar to those described herein. Figures 1 to 4 The memory systems 104, 204, 304, and 404 are described herein. In some embodiments, the mobile computing device 501 may be similar to those described herein. Figure 1 and 2 The computing system 100 and / or computing system 200 described herein.

[0095] Mobile computing device 501 can communicate with base stations (e.g., base station 543-1 or base station 543-N) via one or more communication paths 545-1 to 545-N. Generally, communication paths 545-1 to 545-N can be wireless communication paths that transmit information via electromagnetic radiation at a specific frequency, as described above. In some embodiments, mobile computing device 501 can receive information corresponding to an unknown object 547 from at least one of base stations 543-1 to 543-N via communication paths 545-1 to 545-N.

[0096] For example, base stations 543-1 to 543-N may have previously received information corresponding to the unknown object 547 from other mobile computing devices that have encountered the unknown object 547 and / or from other mobile computing devices that have communicated with base stations 543-1 to 543-N. In such embodiments, mobile computing device 501 (e.g., the processor of mobile computing device 501, such as those described herein in...) Figure 1 and 2 The processors 122 and / or 222 described herein may use reallocated (or pre-allocated) computing resources to perform operations involving data corresponding to the unknown object and / or received information corresponding to the unknown object 547, in order to parse the unknown object 547.

[0097] In some embodiments, when it is determined that an image stored by base stations 543-1 to 543-N resembles an unknown object 547, as part of receiving information corresponding to the unknown object 547 from the base stations, the mobile computing device 501 may receive confidence information about at least one pixel of the image stored by base stations 543-1 to 543-N. For example, if the mobile computing device 501 and / or base stations 543-1 to 543-N determine that an image stored by base stations 543-1 to 543-N or the mobile computing device 501 resembles an unknown object 547, then base stations 543-1 to 543-N or the mobile computing device 501 may generate information corresponding to the confidence levels of base stations 543-1 to 543-N and / or the mobile computing device 501 that one or more pixels of the similar image correspond to the unknown object 547.

[0098] like Figure 5 As shown, the imaging device 521 can receive information (e.g., images and / or live video) related to the unknown object 547. The information can be processed and / or analyzed within the mobile computing device 501, for example, using a memory system 504 residing on the mobile computing device 501. In some embodiments, the information (e.g., images and / or live video) can be processed by the mobile computing device 501 as part of performing navigation assistance operations and / or applications to provide navigation assistance to visually impaired individuals.

[0099] Information relating to the unknown object 547 can be processed by the mobile computing device 501 in conjunction with the execution of one or more applications (e.g., navigation assistance-related applications). As described above, the execution of this application can generate a demanding workload. Therefore, as described herein, information can be selectively written to different memory devices (e.g., in the context of...) based on the characteristics of the workload. Figure 2 The memory devices 223, 225 and / or 227 described herein) are thus written to different media types (e.g., as described herein). Figure 2 The media types 224, 226 and / or 228 described herein are used to optimize the execution of high-priority operations, such as parsing unknown objects that present an impending danger to a visually impaired user of the mobile computing device 501.

[0100] In some embodiments, images and / or videos may be processed and / or analyzed by the mobile computing device 501 during application execution for analysis. Figure 5 The unknown object 547 described herein. In addition, as part of performing the operation of parsing the unknown object, the image and / or video may be processed and / or analyzed by the mobile computing device 501 to detect and / or replace one or more damaged parts (e.g., pixels) of the image and / or video to restore and / or improve the quality of the image and / or video.

[0101] Figure 6 This is a flowchart illustrating various embodiments of the present disclosure corresponding to example methods for navigation assistance for visually impaired individuals. Method 650 can be executed by processing logic, which may include hardware (e.g., a processor, processing device, circuitry, dedicated logic, programmable logic, microcode, device hardware, and / or integrated circuits, etc.), software (e.g., instructions that run or execute on a processor), or a combination thereof. Although shown in a specific order or sequence, the order of processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in different orders, and some processes may be performed in parallel. Furthermore, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0102] At block 651, method 650 may include a processor determining that an image captured by an imaging device coupled to the processor contains an unknown object, the processor being coupled to a first memory device comprising a first type of media and a second memory device comprising a second type of media. The processor may be similar to that described herein. Figure 1 and 2 The processors 122 and 222 are described herein. In this document, the first memory device may be similar to memory devices 123 and 223, while the second memory device may be similar to...Figure 1 and 2 The memory devices 125 and 225 are described herein. Furthermore, in this document, the first type of media may be similar to media types 124 and 224, while the second type of media may be similar to those described herein. Figure 1 and 2 Media types 126 and 226 are described in the text. The imaging device may be similar to those described in this text. Figure 1 and 5 The imaging devices 121 and 521 described herein.

[0103] At block 653, method 650 may include determining, at least in part, that an unknown object represents an unresolvable object within a threshold time period based on an object recognition model executed by a processor. Therefore, in some embodiments, method 650 may include determining that an unknown object is unresolvable within a threshold time period. However, embodiments are not limited thereto and as described above, in some embodiments, method 650 may include determining that an unknown object is unresolvable within a threshold time period based at least in part on a determined confidence level associated with the image.

[0104] At block 655, method 650 may include performing an operation to reallocate computing resources between a first memory device and a second memory device in response to determining that an unknown object is unresolvable within a threshold time period. In some embodiments, method 650 may include determining that the first memory device or the second memory device has a higher bandwidth than the other of the first memory device or the second memory device. In such embodiments, the operation for reallocating computing resources between the first memory device and the second memory device may include reallocating computing resources such that a larger amount of memory with higher bandwidth than the threshold amount is available to perform the operation for resolving the unknown object. However, embodiments are not limited thereto, and in some embodiments, method 650 may include determining that the first memory device or the second memory device has a faster memory access time than the other of the first memory device or the second memory device, and the operation for reallocating computing resources between the first memory device and the second memory device may include reallocating computing resources such that a larger amount of memory with faster memory access time than the threshold amount is available to perform the operation for resolving the unknown object.

[0105] At block 657, method 650 may include a resource whereby the processor writes at least a portion of data associated with an unknown object to a first memory device or a second memory device or both. In some embodiments, at least a portion of the data may be written to a first memory device or a second memory device exhibiting the highest bandwidth, fastest access time, etc.

[0106] At block 659, method 650 may include performing operations involving data corresponding to an unknown object using reallocated computing resources to parse the unknown object. The operations of parsing the unknown object may include performing various machine learning operations to determine the similarity between the unknown object and other objects known to the mobile computing device. However, embodiments are not limited thereto, and in some embodiments, the operations of parsing the unknown object may include replacing one or more pixels of the image of the unknown object with a base station and / or other mobile computing device in a polling region to determine whether the unknown object is recognized by the base station and / or other mobile computing device.

[0107] As described above, the processor, the first memory device, and the second memory device may reside in a mobile computing device (e.g., as described herein). Figure 5 On the mobile computing device 501 described herein. In such embodiments, method 650 may include determining that an unknown object presents an impending danger to the user of the mobile computing device, and updating route information associated with the mobile computing device to guide the user of the mobile computing device away from the unknown object. However, embodiments are not limited thereto, and in some embodiments, method 650 may include determining that the unknown object cannot be resolved within the time corresponding to the user of the mobile computing device coming into contact with the unknown object, and updating route information associated with the mobile computing device to guide the user of the mobile computing device away from the unknown object.

[0108] In some embodiments, method 650 may include determining that the processor is retrieving data from a first base station (e.g., as described herein). Figure 5 The base station 543-1 described herein receives network coverage and determines that the processor has moved to a different geographical location and is receiving data from a second base station (e.g., in this document). Figure 5 The base station 543-N described herein receives network coverage. In such embodiments, method 650 may include performing operations to reallocate computing resources between a first memory device and a second memory device in response to determining that the processor is receiving network coverage from a second base station.

[0109] Method 650 may further include generating a vibration pattern corresponding to navigation in the presence of an unknown object, and / or a voice-assisted navigation update corresponding to navigation in the presence of an unknown object. In some embodiments, a vibration pattern may be generated such that a portion of the vibration of the mobile computing device and / or the voice-assisted navigation update is projected by the mobile computing device to indicate to the user of the mobile computing device the direction in which they should move to avoid contact with the unknown object.

[0110] As described above, the first memory device or the second memory device may be a non-persistent memory device, and the other of the first memory device or the second memory device may be a persistent memory device. In some embodiments, the processor, the first memory device, and the second memory device may reside in a mobile computing device (e.g., as described herein). Figure 5 The mobile computing device 501 described herein. In such embodiments, method 650 may include being determined, written, and initiated by the processor without control signals generated outside the mobile computing device. Embodiments are not limited thereto, and in some embodiments, method 650 may include writing at least a portion of data associated with the workload to another of the first or second memory devices as part of operations to optimize the parsing of unknown objects and / or optimize the battery consumption of the mobile computing device.

[0111] While specific embodiments have been shown and described herein, those skilled in the art will understand that arrangements calculated to achieve the same results may replace the shown specific embodiments. This disclosure is intended to cover modifications or variations of one or more embodiments of this disclosure. It should be understood that the above description has been carried out illustratively and not restrictively. Combinations of the above embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art upon review of the above description. The scope of one or more embodiments of this disclosure includes other applications in which the above structures and processes are used. Therefore, the scope of one or more embodiments of this disclosure should be determined with reference to the appended claims together with the full scope of the equivalents given by such claims.

[0112] In the foregoing detailed embodiments, some features are grouped together in a single embodiment for the purpose of simplifying this disclosure. This approach of the disclosure should not be construed as reflecting an intention that the disclosed embodiments must use more features than expressly stated in each claim. In fact, as reflected in the appended claims, the subject matter of the invention lies in less than all the features of a single disclosed embodiment. Therefore, the appended claims are hereby incorporated into the detailed embodiments, wherein each claim is an independent embodiment.

Claims

1. A method (650) for navigation assistance for visually impaired persons, comprising: The processor (122, 222) determines that the image captured by the imaging device (121) coupled to the processor (122, 222) contains an unknown object (547), the processor being coupled to a first memory device (123, 223) comprising a first type of media (124, 224) and a second memory device (125, 225) comprising a second type of media (126, 226); At least a portion of the unknown object (547) is determined to be an unresolvable object within a threshold time period based at least in part on an object identification model executed by the processors (122, 222); In response to determining that the unknown object (547) is unresolvable within the threshold time period, an operation is performed to reallocate computing resources between the first memory device (123, 223) and the second memory device (125, 225); The processors (122, 222) write at least a portion of the data associated with the unknown object (547) to the reallocated computing resources of the first memory device (123, 223) or the second memory device (125, 225) or both; and The reallocated computing resources are used to perform operations involving the data corresponding to the unknown object (547) to parse the unknown object (547).

2. The method of claim 1, further comprising determining, at least in part, that the unknown object is unresolvable during the threshold time period based on a determined confidence level associated with the image.

3. The method according to any one of claims 1 to 2, further comprising: It is determined that one of the first memory device or the second memory device has a higher bandwidth than the other of the first memory device or the second memory device, wherein The operation for reallocating computing resources between the first memory device and the second memory device includes reallocating the computing resources such that a larger amount of memory with the higher bandwidth is available to perform the operation for resolving the unknown object, or It is determined that one of the first memory device or the second memory device has a faster memory access time than the other of the first memory device or the second memory device, wherein The operation for reallocating computing resources between the first memory device and the second memory device includes reallocating the computing resources such that a greater than a threshold amount of memory with the faster memory access time is available to perform the operation for resolving the unknown object.

4. The method according to any one of claims 1 to 2, wherein the processor, the first memory device, and the second memory device reside on the mobile computing device (501), and wherein the method further comprises: It is determined that the unknown object presents an impending danger to the user of the mobile computing device, or It is determined that the unknown object cannot be resolved within a threshold time period corresponding to the user's contact with the unknown object on the mobile computing device (501); and Update the route information associated with the mobile computing device (501) to guide the user of the mobile computing device away from the unknown object.

5. The method according to any one of claims 1 to 2, further comprising: It is determined that the processor is receiving network coverage from the first base station (543-1); It was determined that the processor had been moved to a different geographical location and was receiving network coverage from the second base station (543-N); and The operation is performed in response to determining that the processor is receiving network coverage from the second base station (543-N) to reallocate computing resources between the first memory device and the second memory device.

6. The method according to any one of claims 1 to 2, further comprising generating a vibration pattern corresponding to navigation in the presence of the unknown object, or a voice-assisted navigation update corresponding to navigation in the presence of the unknown object, or both.

7. A navigation aid for visually impaired individuals, comprising: A first memory device (123, 223) includes a first type of media (124, 224); The second memory device (125, 225) includes a second type of media (126, 226); and Processors (122, 222) coupled to the first memory device (123, 223) and the second memory device (125, 225), wherein the processors (122, 222) will: It is determined that the image captured by the imaging device (121) coupled to the processor (122, 222) contains an unknown object (547); The unknown object (547) is determined to be unresolvable within a threshold time period, at least in part, based on a determined confidence level associated with the image; In response to determining that the unknown object (547) is unresolvable within the threshold time period, an operation is performed to reallocate computing resources between the first memory device (123, 223) and the second memory device (125, 225); Write at least a portion of the data associated with the unknown object (547) to the reallocated computing resources of the first memory device (123, 223) or the second memory device (125, 225) or both; and The reallocated computing resources are used to perform operations involving the data corresponding to the unknown object (547) to parse the unknown object (547).

8. The device of claim 7, wherein the processor determines, at least in part, that the unknown object is indistinguishable during the threshold time period based on an object recognition model executed by the processor.

9. The device according to claim 7, wherein: The first memory device or the second memory device is a NAND flash memory device (333, 433), which includes a set of single-level memory cells (SLC) (335, 435) and a set of multi-level memory cells (MLC) (337, 437), and The processor writes the data corresponding to the unknown object into the set of single-level memory cells (SLCs) (335, 435) as part of the operation of parsing the unknown object.

10. The device of claim 7, wherein the processor, the first memory device, and the second memory device reside on the mobile computing device (501), and wherein the processor will: It is determined that the mobile computing device (501) has undergone a handover from the first base station (543-1) to the second base station (543-N); and In response to the determination that the mobile computing device has undergone the handover, the operation is performed to reallocate computing resources between the first memory device and the second memory device.

11. The device according to any one of claims 7 to 9, wherein the processor, the first memory device, and the second memory device reside on the mobile computing device (501), and wherein the processor will: Determining that the unknown object presents an impending danger to the user of the mobile computing device (501) or determining that the unknown object cannot be resolved within a threshold time period corresponding to the user's contact with the unknown object, or both; and The user of the mobile computing device (501) is guided away from the unknown object by updating the route information associated with the mobile computing device (501).

12. The device according to any one of claims 7 to 9, wherein the processor executes instructions corresponding to one or more machine learning operations as part of performing the operation involving the data corresponding to the unknown object to parse the unknown object.

13. A navigation assistance system for visually impaired individuals, comprising: A mobile computing device (501) includes a processor (122, 222), a first memory device (123, 223), a second memory device (125, 225), and a third memory device (227). and An imaging device (121), residing on the mobile computing device (501) and coupled to the processors (122, 222), wherein the processors (122, 222) will: Receive an image captured by the imaging device 121, the image containing more than a threshold number of unidentifiable pixels; The captured image is classified as an image containing an unknown object (547) based on the fact that the image contains more than the threshold number of unidentifiable pixels; Based at least in part on an object recognition model executed by the processor (122, 222) or a certain confidence level associated with the captured image, or both, it is determined that at least a portion of the unrecognizable pixels are unresolvable during a first threshold time period; It is determined that the unknown object (547) poses an impending danger to the user of the mobile computing device (501); In response to determining that at least a portion of the unidentifiable pixel is unresolvable during the first threshold time period or determining that the unknown object (547) poses an impending danger to the user of the mobile computing device (501), or both, computing resources are reallocated in the first memory device (123, 223), the second memory device (125, 225), or the third memory device (227), or any combination thereof. The reallocated computing resources are used to perform operations involving at least the portion of the unidentifiable pixels corresponding to the unknown object (547) to resolve the unknown object (547); and Within a second threshold time period, determine whether the operation used to parse the unknown object (547) was successful.

14. The system of claim 13, wherein in response to determining that the operation for parsing the unknown object was successful within the second threshold time period, the processor notifies the user of the mobile computing device to parse the unknown object.

15. The system of claim 13, wherein in response to determining that the operation for resolving the unknown object is unsuccessful within the second threshold time period, the processor updates the route information associated with the mobile computing device to guide the user of the mobile computing device away from the unknown object.

16. The system according to claim 13, wherein: At least one of the first memory device, the second memory device, or the third memory device includes a flash memory device (333, 433). At least one of the first memory device, the second memory device, or the third memory device includes a dynamic random access memory device (331, 431), and At least one of the first memory device, the second memory device, or the third memory device includes a variable resistance memory device (439).

17. The system according to any one of claims 13 to 16, wherein the processor will: It is determined that one of the first memory device, the second memory device, or the third memory device exhibits higher bandwidth than the other or both of the first memory device, the second memory device, or the third memory device. and Performing the operation to reallocate computing resources among the first memory device, the second memory device, or the third memory device, or any combination thereof, such that: The memory containing a higher bandwidth than the threshold can be used to perform the operation for parsing the unknown object, or A greater-than-threshold amount of memory exhibiting faster memory access times can be used to perform the operation for resolving the unknown object; and The operation involving the unidentifiable pixels corresponding to the unknown object is performed using the reallocated computing resources, to parse the unknown object using the first memory device, the second memory device, or the third memory device, or any combination thereof, which exhibits the higher bandwidth to perform the operation for parsing the unknown object or the faster memory access time to perform the operation for parsing the unknown object, or both.

18. The system according to any one of claims 13 to 16, wherein the processor will: It is determined that the mobile computing device has undergone a handover from the first base station (543-1) to the second base station (543-N); and In response to the determination that the mobile computing device has undergone the handover, the operation is performed to reallocate computing resources between the first memory device and the second memory device.

19. The system according to any one of claims 13 to 16, wherein the processor generates a vibration pattern corresponding to navigation in the presence of the unknown object, or a voice-assisted navigation update corresponding to navigation in the presence of the unknown object, or both.

20. The system of any one of claims 13 to 16, wherein the processor performs the operation relating to the unidentifiable pixel corresponding to the unknown object to resolve the unknown object as part of executing instructions to assist a visually impaired user of the mobile computing device.

Images