Autonomous vehicle object detection

Abstract

This invention describes methods, systems, and apparatus related to object detection in autonomous vehicles. One method may include: receiving an indication from the autonomous vehicle that it has entered a network coverage area generated by a base station; and in response to receiving the indication, performing operations to reallocate computing resources among a plurality of different types of memory devices associated with the autonomous vehicle. The method may further include: capturing data corresponding to unknown objects located within the line of sight of the autonomous vehicle; and using the reallocated computing resources to perform operations relating to the data corresponding to the unknown objects to classify the unknown objects.

Description

Technical Field

[0001] This disclosure generally relates to semiconductor memories and methods, and more specifically, to apparatus, systems and methods for detecting objects in autonomous vehicles. Background Technology

[0002] Memory devices are typically provided as internal semiconductor integrated circuits in computers or other electronic systems. Many different types of memory exist, including volatile and non-volatile memory. Volatile memory requires power to maintain its data (e.g., host data, error data, etc.) and includes random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), and thyristor random access memory (TRAM), etc. Non-volatile memory provides persistent data by retaining the stored data when no power is applied and includes NAND flash memory, NOR flash memory, and resistive variable memory, such as phase-change random access memory (PCRAM), resistive random access memory (RRAM), and magnetoresistive random access memory (MRAM), such as spin torque transfer random access memory (STT RAM), etc.

[0003] A memory device may be coupled to a host computer (e.g., a host computing device) to store data, commands, and / or instructions for use by the host computer or electronic system during operation. For example, during the operation of a computing or other electronic system, data, commands, and / or instructions may be transferred between the host computer and the memory device. Summary of the Invention

[0004] On one hand, this disclosure relates to a method for object detection in an autonomous vehicle, comprising: receiving an indication that the autonomous vehicle has entered a network coverage area generated by a base station; in response to receiving the indication, performing operations to reallocate computing resources among a plurality of different types of memory devices associated with the autonomous vehicle; capturing data corresponding to an unknown object placed within the line of sight of the autonomous vehicle; and using the reallocated computing resources to perform operations relating to the data corresponding to the unknown object to classify the unknown object.

[0005] On the other hand, this disclosure relates to an apparatus for object detection in an autonomous vehicle, comprising: an autonomous vehicle including: a first memory device including a first type of media; a second memory device including a second type of media; an imaging device; and a processing unit coupled to the first memory device, the second memory device, and the imaging device, wherein the processing unit is configured to: receive an indication that the device has entered a network coverage area generated by a base station; in response to receiving the indication, perform operations to reallocate computing resources between the first memory device and the second memory device; cause the imaging device to capture at least one image corresponding to an unknown object placed within the line of sight of the imaging device; and use the reallocated computing resources to perform operations relating to the captured at least one image corresponding to the unknown object to classify the unknown object.

[0006] On the other hand, this disclosure relates to a non-transitory computer-readable storage medium for object detection of autonomous vehicles, comprising instructions that, when executed by a processing unit, cause the processing unit to: receive, via a radio frequency integrated circuit (RFIC) coupled to the processing unit, an instruction corresponding to an autonomous vehicle entering a network coverage area generated by a base station; perform operations in response to receiving the instruction to reallocate computing resources between a first memory device and a second memory device associated with the autonomous vehicle; cause an imaging device associated with the autonomous vehicle to capture at least one image corresponding to an unknown object positioned within the line of sight of the imaging device; and cause operations relating to the captured at least one image corresponding to the unknown object to be performed using the reallocated computing resources to classify the unknown object. Attached Figure Description

[0007] Figure 1 This is a functional block diagram of a device including a host and a memory device according to several embodiments of the present disclosure.

[0008] Figure 2 This is another functional block diagram of a computing system comprising a device including a host and a memory system, according to several embodiments of the present disclosure.

[0009] Figure 3 This is a functional block diagram in the form of a device including a memory system according to several embodiments of the present disclosure.

[0010] Figure 4 This is another functional block diagram in the form of a device including a memory system according to several embodiments of the present disclosure.

[0011] Figure 5This is a schematic diagram of an autonomous vehicle including an electronic control unit according to several embodiments of the present disclosure.

[0012] Figure 6 This is a flowchart illustrating an instance method corresponding to autonomous vehicle object detection according to several embodiments of the present disclosure.

[0013] Figure 7 This is a schematic diagram illustrating a non-transitory computer-readable storage medium according to several embodiments of the present disclosure. Detailed Implementation

[0014] Methods, systems, and apparatus related to object detection in autonomous vehicles are described. One method may include: receiving an indication from the autonomous vehicle that it has entered a network coverage area generated by a base station; and in response to receiving the indication, performing operations to reallocate computing resources among multiple different types of memory devices associated with the autonomous vehicle. The method may further include: capturing data corresponding to unknown objects positioned within the line of sight of the autonomous vehicle; and using the reallocated computing resources to perform operations involving the data corresponding to the unknown objects to classify the unknown objects.

[0015] As autonomous vehicles (e.g., cars, trucks, buses, motorcycles, mopeds, all-terrain vehicles, military vehicles, tanks, etc., in which at least a portion of decision-making and / or control of vehicle operation is controlled by computer hardware and / or software, rather than by a human operator) become increasingly prevalent, the safety of such vehicles must be addressed. While various methods exist to mitigate the dangers associated with autonomous vehicles and thus improve their safety, limitations in the computational resources (e.g., computer hardware and software) controlling autonomous vehicles, coupled with the constantly changing environment in which they operate, make such improvements challenging.

[0016] For example, the speed at which autonomous vehicles can accurately determine objects (e.g., traffic signs, other vehicles on the road, etc.) within or outside their path may, in some methods, be limited by the speed and / or accuracy at which resources can ingest and process received data during the operation of the autonomous vehicle. These limitations can be further amplified when the autonomous vehicle detects unknown objects (e.g., objects that have not been previously detected, analyzed, or otherwise unidentified by the autonomous vehicle for various reasons, such as the object being a known object that has been contaminated in some way). The terms "object" or "unknown object" are used interchangeably herein with the terms "obstacle" or "unknown obstacle," respectively.

[0017] For example, autonomous vehicles may easily recognize stop signs on the roadside and control their operation accordingly. However, if a stop sign is partially covered by snow, dirt, stickers, graffiti, or otherwise obscured, the autonomous vehicle may be unable to recognize it and control its operation accordingly. In this non-limiting example, the inability of an autonomous vehicle to detect and recognize a stop sign due to obscuration can lead to a dangerous and unsafe situation, which in the worst case could result in an accident that endangers lives and could cause death.

[0018] However, the examples of unknown objects that autonomous vehicles may encounter during operation, as envisioned in this disclosure, are not limited to those described above. Therefore, the embodiments described herein can be applied to a wide range of objects that can be classified as unknown objects or that have been contaminated to the point of being unknown objects, such as streetlights, street signs, construction debris, debris generated by weather events and / or debris generated by human events (e.g., debris from a car accident that may have recently settled on the road, debris that fell from an airplane and thus recently settled on the road, etc.).

[0019] As described in more detail herein, aspects of this disclosure allow for the timely and accurate resolution of such unknown objects by purposefully reallocating computing resources (e.g., processing and / or memory resources) available to autonomous vehicles, such that the most efficient (e.g., fastest, most accurate, etc.) computing resources can be used as needed to process information about unknown objects to resolve them and allow the autonomous vehicle to navigate safely in the presence of unknown objects. As used herein, the term "resolve / resolution," depending on the context, generally refers to identifying and / or determining what an unknown object is. For example, if the unknown object is a damaged stop sign, then resolving the unknown object generally means determining that the unknown object is indeed a stop sign. In some embodiments, objects can be classified based on their resolution. For example, objects can be classified as street signs, traffic lights, debris, etc., after being resolved.

[0020] To facilitate embodiments of this disclosure, autonomous vehicles may reallocate or pre-allocate computing resources based on traffic sequence prediction modeling and / or in response to a determination that the autonomous vehicle will move from an area receiving network coverage from a base station to an area receiving network coverage from different base stations. As used herein, the term "network coverage," particularly in the context of network coverage from base stations, generally refers to a geographical area characterized by the presence of electromagnetic radiation (e.g., waves having a specific frequency range associated therewith) generated by a base station. As used herein, "base station" generally refers to equipment that generates and receives electromagnetic radiation within a specific frequency range and facilitates the transmission of data or other information between the base station and computing devices (e.g., smartphones, autonomous vehicles, etc.) within the base station's network coverage area. Several non-limiting examples of the frequency ranges that a base station can generate and receive may include 700MHz to 2500MHz (in the case of 4G base stations) or 28GHz to 39GHz (in the case of 5G base stations).

[0021] In embodiments where traffic sequence prediction modeling is used as a part of the reallocation or pre-allocation of computing resources, the autonomous vehicle may execute instructions associated with one or more traffic sequence prediction operations to determine the traffic volume ahead of the road on which the autonomous vehicle operates. If the traffic sequence prediction operation determines that there is heavy traffic at a certain location ahead of the road, the autonomous vehicle may reallocate or pre-allocate computing resources such that the fastest available computing resources are available before the autonomous vehicle encounters the heavy traffic. Similarly, in embodiments where determining that the autonomous vehicle will move from an area receiving network coverage from one base station to an area receiving network coverage from different base stations is part of the reallocation or pre-allocation of computing resources, the autonomous vehicle may reallocate or pre-allocate computing resources such that the fastest available computing resources are available before the autonomous vehicle enters an area receiving network coverage from different base stations.

[0022] In the following detailed description of this disclosure, reference is made to the accompanying drawings, which form a part of this disclosure, and one or more embodiments of this disclosure are illustrated by way of illustration in the drawings. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments of this disclosure, and it should be understood that other embodiments may be utilized and process changes, electrical changes and / or structural changes may be made without departing from the scope of this disclosure.

[0023] As used herein, particularly with respect to reference numerals in the figures, the indicators “N” and “M” may include several specific features as specified herein. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a” and “described” may include both singular and plural indicators unless the context clearly indicates otherwise. Additionally, “a number,” “at least one,” and “one or more” (e.g., a number of memory banks) may refer to one or more memory banks, while “more” is intended to refer to more than one such thing.

[0024] Furthermore, the word "can / may" is used throughout this application in a permissive sense (i.e., possible, able) rather than a mandatory sense (i.e., required). The term "comprising" and its derivatives mean "including but not limited to". The terms "coupled" and "coupled" mean a direct or indirect physical connection, or for accessing and moving (transmitting) commands and / or data, as the context requires. The terms "data" and "data value" are used interchangeably herein and may have the same meaning, as the context requires.

[0025] The diagrams in this document follow a numbering convention, where the first one or a few digits correspond to the diagram number, and the remaining digits identify the elements or components within the diagram. Similar elements or components between different diagrams can be identified using similar digits. For example, 104 could refer to... Figure 1 Component "04" in the text, and similar components in Figure 2 The element may be designated as 204. In this document, a group or plurality of similar elements or components may generally be referred to by a single element number. For example, multiple reference elements, such as elements 126-1 to 126-N (or, alternatively, 126-1, ..., 126-N), may be collectively referred to as 126. It will be understood that elements shown in the various embodiments herein may be added, interchanged, and / or eliminated to provide several additional embodiments of this disclosure. Furthermore, the scale and / or relative dimensions of the elements provided in the figures are intended to illustrate certain embodiments of this disclosure and should not be construed as limiting.

[0026] Figure 1 This is a functional block diagram of a computing system 100 comprising a device including a host 102 and a memory system 104, according to several embodiments of the present disclosure. In some embodiments, the host 102 and / or memory system 104 may be part of an electronic control unit (ECU) 101 (e.g., an electronic control unit of an autonomous vehicle). As used herein, "device" may refer to, for example, but not limited to, any of a variety of structures or combinations thereof, such as a circuit or circuit system, a die or several dies, a module or several modules, a device or several devices or systems or several systems. In some embodiments, the computing system 100 may be an autonomous vehicle (e.g., as described herein) Figure 5 This refers to the autonomous vehicle 541 described herein. For example, the computing system 100 may reside on the autonomous vehicle. In such embodiments, the computing system 100 can control the operation of the autonomous vehicle by controlling, for example, the acceleration, braking, steering, stopping, etc.

[0027] As used herein, the term "resides on" means that something is physically located on a particular component. For example, computing device 100 residing on an autonomous vehicle means that computing system 100 is physically coupled to or physically located within an autonomous vehicle. The term "resides on" may be used interchangeably herein with other terms such as "deployed on" or "located on".

[0028] Memory system 104 may include several different memory devices 123, 125 (and / or herein) Figure 2 As described in section 227), it may include one or more different media types 124, 126 (and / or this article). Figure 2 (See 228 in the description). Different memory devices 123, 125 and / or 227 may include one or more memory modules (e.g., single in-line memory modules, dual in-line memory modules, etc.).

[0029] Memory system 104 may include volatile memory and / or non-volatile memory. In several embodiments, memory system 104 may include a multi-chip device. The multi-chip device may include several different memory devices 123, 125, and / or 227, which may include several different memory types and / or memory modules. For example, the memory system may include non-volatile or volatile memory on any type of module. Figure 1 As shown, the memory system 104 may include a controller 120, which may include a processing unit 122. Each of the components (e.g., ECU 101, host 102, controller 120, processing unit 122, and / or memory devices 123, 125) may be individually referred to herein as a “device”.

[0030] Memory system 104 may provide main memory for computing system 100, or may be used as additional memory and / or storage devices throughout computing system 100. Memory system 104 may include one or more memory devices 123, 125, which may contain volatile and / or non-volatile memory cells. For example, at least one of memory devices 123, 125 may be a flash array having a NAND architecture. Furthermore, at least one of memory devices 123, 125 may be a dynamic random access array of memory cells. Embodiments are not limited to a specific type of memory device. For example, memory system 104 may include RAM, ROM, DRAM, SDRAM, PCRAM, RRAM, and / or flash memory (e.g., NAND and / or NOR flash memory devices), etc.

[0031] However, the embodiments are not limited thereto, and the memory system 104 may include other non-volatile memory devices 123, 125, such as non-volatile random access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), "emerging" memory devices, such as variable resistance (e.g., 3-D crosspoint (3D XP)) memory devices, memory devices containing self-select memory (SSM) cell arrays, or any combination thereof.

[0032] Variable resistance memory devices can perform bit storage based on changes in bulk resistance, combined with stackable cross-grid data access arrays. Furthermore, compared to many flash-based memories, variable resistance non-volatile memories can perform in-situ write operations, where non-volatile memory cells can be programmed without prior erasing. Compared to flash-based memories and variable resistance memories, self-selecting memory cells can comprise memory cells with a single chalcogenide material, which serves as both the switch and storage element of the memory cell.

[0033] As in Figure 1As shown herein, memory devices 123 and 125 comprise different types of memory devices. For example, memory device 125 may be a 3D XP memory device or a NAND memory device, and memory device 123 may be a volatile memory device, such as a DRAM device, and vice versa. That is, memory devices 123 and 125 may comprise different media types 124 and 126. However, embodiments are not limited thereto, and memory devices 123 and 125 may comprise any type of memory device, provided that at least two of memory devices 123 and 125 comprise different media types 124 and 126. As used herein, "media type" generally refers to the type of memory cell architecture corresponding to memory devices 123 and 125. For example, one of media types 124 and 126 may correspond to a memory cell array comprising at least one capacitor and at least one transistor, while the other of media types 124 and 126 may comprise a floating gate metal-oxide-semiconductor field-effect transistor array. In some embodiments, at least one of media types 124, 126 may include an array of resistive memory cells configured to perform bit storage based on changes in the volume resistance associated with the resistive memory cells.

[0034] As in Figure 1 As illustrated, host 102 may be coupled to memory system 104. In several embodiments, memory system 104 may be coupled to host 102 via one or more channels (e.g., channel 103). Figure 1 In this configuration, memory system 104 is coupled to host 102 via channel 103, and host 102 may also be coupled to controller 120 and / or processing unit 122 of memory system 104. Controller 120 and / or processing unit 122 are coupled to memory devices 123 and 125 via channels 105 and 107. In some embodiments, each of memory devices 123 and 125 is coupled to controller 120 and / or processing unit 122 via one or more corresponding channels 105 and 107, such that each of memory devices 123 and 125 can receive messages, commands, requests, protocols, or other signaling conforming to the type of memory device 123 or 125 coupled to controller 120 (e.g., messages, commands, requests, protocols, or other signaling conforming to media types 124 and 126 of memory devices 123 and 125).

[0035] ECU 101 may further include an imaging device 121. The imaging device 121 may be communicatively coupled to host 102 and / or memory device 104 (e.g., to controller 120 and / or processing unit 122). The imaging device 121 may be a camera, ultrasonic device, ultrasound device, stereo imaging device, infrared imaging device, or other imaging device that can capture data containing images or image streams (e.g., streaming video and / or “real-time streaming video”) in real time and transmit information corresponding to the images and / or image streams to computing system 100. Generally, the imaging device may be any mechanical, digital, or electronic observation device; a still camera; a video camera; a cinema camera; or an instrument, apparatus, or format capable of recording, storing, or transmitting images, video, and / or information.

[0036] As used herein, the term "real-time streaming video" and variations thereof generally refer to a sequence of images captured and processed, reproduced, and / or broadcast simultaneously (or nearly simultaneously). In some embodiments, alternatively herein, "real-time streaming" video may be referred to as "data captured by an imaging device" or "data captured from an imaging device." Furthermore, as used herein, the term "streaming video" and variations thereof generally refer to a sequence of images captured by an imaging device and subsequently processed, reproduced, and / or broadcast. In some embodiments, alternatively herein, "streaming" video may be referred to as "data captured by an imaging device" or "data captured from an imaging device."

[0037] Generally, such data captured by imaging devices (e.g., images, image streams, and / or "real-time streaming" video) can be processed and / or analyzed by components of ECU 101 as part of object detection and / or object identification for the purpose of safe operation of the autonomous vehicle. Object detection and / or object identification refers to the process performed by the autonomous vehicle (or by its electrical system, such as ECU 101) to distinguish various objects that may be in or near the path of the autonomous vehicle. Under ideal operating conditions, such data can be compared with a database of known objects, and information derived from said comparison can be used to guide how the autonomous vehicle operates (e.g., whether to decelerate, accelerate, stop, etc.).

[0038] However, as mentioned above, objects that cannot be identified by such comparisons may appear in or near the path of the autonomous vehicle (e.g., within the line of sight of imaging device 121) for various reasons. In many current methods, these unknown objects may be difficult to distinguish or identify, especially in a timely and accurate manner. In contrast, by utilizing aspects of this disclosure, the identification of unknown objects can be achieved quickly and accurately, thereby improving the operation and safety of autonomous vehicles. For example, embodiments herein may allow pre-allocation of computing resources to selectively process workloads involving images and / or videos corresponding to those captured by imaging device 121, such that workloads corresponding to the execution of applications involving said images and / or videos are allocated to memory devices 123, 125, 227 exhibiting specific characteristics to optimize the performance of memory system 104, enabling accurate detection and identification of unknown objects via autonomous vehicles.

[0039] In some embodiments, imaging device 121 may capture data including images of objects (known or unknown) and / or obstacles (known or unknown) used by the autonomous vehicle, such as images and / or streaming video (e.g., real-time streaming video). For example, in some embodiments, the images and / or streaming video captured by imaging device 121 may include images of obscured traffic signs, debris accumulated on or around the path of the autonomous vehicle, etc. Such images and / or streaming video may be captured by imaging device 121 and processed locally within ECU 101 and / or memory system 104 as part of the operation of distinguishing unknown objects captured by imaging device 121.

[0040] ECU 101 may further include a radio frequency integrated circuit (RFIC) 111. As used herein, the term "RFIC" generally refers to an electrical integrated circuit operating in a frequency range suitable for wireless transmission. In some embodiments, RFIC 111 may facilitate autonomous vehicles (e.g., as described herein). Figure 5 The autonomous vehicles described in this paper 541), base stations (e.g., in this paper) Figure 5 The communication between the base station 543 described herein and / or other autonomous vehicles operating on the road or street where the autonomous vehicle containing RFIC 111 is operating. In some embodiments, RFIC 111 may be communicatively coupled to controller 120 and / or processing unit 122 via a Double Data Rate (DDR) interface such as DRR4, DDR5, etc.

[0041] Additionally, ECU 101 may further include various sensors not shown to avoid confusion with the diagrams. For example, ECU 101 may include inertial sensors, radar sensors, LIDAR sensors, etc., which can be used to assist the navigation and operation of autonomous vehicles.

[0042] The host 102 may be a host system, such as a personal laptop computer, desktop computer, digital camera, smartphone, memory card reader / or Internet of Things (IoT) enabled device, and various other types of host. However, in some embodiments, the host 102 includes one or more central processing units that execute instructions to control the operation of the autonomous vehicle.

[0043] Those skilled in the art will understand that "processor" can mean one or more processors, such as in a parallel processing system, several coprocessors, etc. System 100 may include a single integrated circuit, or one or more of the host 102, memory system 104, control circuitry system 120, and / or memory devices 123, 125, and / or 227 may be on the same integrated circuit. For example, computing system 100 may be a server system and / or a high-performance computing (HPC) system and / or a portion thereof. Although Figure 1 The examples shown illustrate systems with a von Neumann architecture, but embodiments of this disclosure can be implemented in non-von Neumann architectures, which may not include one or more components typically associated with a von Neumann architecture (e.g., CPU, ALU, etc.).

[0044] Memory system 104 may include controller 120, which may include processing unit 122. Processing unit 122 may be provided in the form of an integrated circuit, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a reduced instruction set computing device (RISC), an advanced RISC machine, a system-on-a-chip, or other combination of hardware and / or circuitry configured to perform the operations described in more detail herein. In some embodiments, processing unit 122 may include one or more processors (e.g., processing devices, coprocessors, etc.).

[0045] Processing unit 122 can perform operations to monitor and / or determine characteristics of workloads running on memory system 104 (e.g., workloads associated with the detection and differentiation of unknown objects). These characteristics may include information such as bandwidth consumption, memory resource consumption, access frequency (e.g., whether the data is hot or cold), and / or power consumption during workload execution. Processing unit 122 can control the writing of at least a portion of data stored in one memory device 123, 125 to different memory devices 123, 125 to optimize workload execution and balance the workload across the different memory devices 123, 125, allowing for rapid and accurate differentiation of unknown objects detected by autonomous vehicles.

[0046] In a non-limiting instance, the device (e.g., computing system 100) may include components residing in an autonomous vehicle (e.g., this document). Figure 5 The device includes a first memory device 123 on the autonomous vehicle 542 described herein, comprising a first type of media 124, and a second memory device 125 residing on the autonomous vehicle, comprising a second type of media 126. The device may further include an imaging device 121 residing on the autonomous vehicle and a processing unit 122 residing on the autonomous vehicle. The processing unit 122 may be coupled to the first memory device 123, the second memory device 125, and the imaging device 121. The processing unit 122 may receive information that the device has entered a base station (e.g., as described herein). Figure 5 The network coverage area indication generated by base station 543 as described above. As described above, in some embodiments, RFIC 111 facilitates communication between autonomous vehicles and base stations.

[0047] Processing unit 122 may perform operations in response to receiving an instruction to reallocate (or pre-allocate, as described herein) computing resources between first memory device 123 and second memory device 125. Continuing this example, processing unit 122 may cause imaging device 121 to capture an unknown object (e.g., as described herein) positioned within the line of sight of imaging device 121. Figure 5 At least one image of the unknown object 547 described herein. In some embodiments, the processing unit 122 may use reallocated computing resources to perform operations involving at least one captured image corresponding to the unknown object to classify the unknown object.

[0048] Processing unit 122 executes instructions to perform a traffic sequence prediction operation to determine traffic sequence prediction modeling information associated with the operating area of ​​the autonomous vehicle, and performs operations to reallocate computing resources between a first memory device 123 and a second memory device 125, at least in part based on the determined traffic sequence prediction modeling information. As used herein, the term "traffic sequence prediction operation" generally refers to performing an operation to estimate, determine, or otherwise predict the quantity of objects (known or unknown) that the autonomous vehicle will encounter in the future. A traffic sequence prediction operation may include performing a deep learning algorithm and / or receiving information from other autonomous vehicles on the road and / or from a base station communicating with the autonomous vehicle. A traffic sequence prediction operation may be performed to determine the probability that the autonomous vehicle will encounter a quantity of objects (known or unknown) greater than or less than a threshold within a given threshold time period. For example, a traffic sequence operation may be performed to determine whether the road is clear (e.g., minimal traffic and / or objects on the road within a few miles or kilometers), congested (e.g., heavy traffic ahead on the road within a few miles and / or kilometers and / or a large number of objects in or near the road), or somewhere in between.

[0049] If the traffic sequence prediction operation determines that a road (or an area of ​​road ahead of the autonomous vehicle) is congested, then processing unit 122 may pre-allocate processing resources available for the first memory device 123 and the second memory device 125 in response to the traffic sequence prediction operation indicating that the autonomous vehicle will encounter an object quantity greater than a threshold object quantity. Furthermore, in some embodiments, if the traffic sequence prediction operation determines that a road (or an area of ​​road ahead of the autonomous vehicle) is congested, then processing unit 122 may increase the data collection rate (e.g., the rate at which imaging device 121 collects images or videos) and / or increase the operating frequency of memory devices 123, 125 and / or controller 120.

[0050] In some embodiments, processing unit 122 may pre-allocate processing resources available for the first memory device 123 and the second memory device 125, such that the memory devices with higher performance characteristics (e.g., higher bandwidth, higher accuracy, faster performance, etc.) have sufficient free space to store images (or videos) corresponding to unknown objects in order to process / analyze the unknown objects as quickly and accurately as possible. That is, in some embodiments, processing unit 122 may allocate processing and / or memory resources available for the autonomous vehicle when an unknown object is expected to be encountered, such that the processing and / or memory resources can be used to immediately identify the unknown object.

[0051] Processing unit 122 may use pre-allocated processing resources to perform operations involving captured images (or videos) corresponding to unknown objects to classify the unknown objects. As described above, classifying unknown objects may include identifying unknown objects to determine what the unknown objects are. Identifying unknown objects may include repairing missing or smudged pixels in the image corresponding to the unknown object, performing machine learning or deep learning algorithms, and / or receiving information corresponding to the unknown object from other autonomous vehicles and / or from base stations communicating with the autonomous vehicles, as well as other techniques.

[0052] In some embodiments, the processing unit 122 may be derived from a base station (e.g., this document). Figure 5 The base station 543 described herein receives information corresponding to an unknown object and causes the reassigned processing resources to perform operations involving data corresponding to the unknown object and received information corresponding to the unknown object in order to classify the unknown object.

[0053] As mentioned above, the first memory device 123 or the second memory device 125 may be a non-persistent (e.g., volatile) memory device, and the other of the first memory device 123 or the second memory device 125 may be a persistent (e.g., non-volatile) memory device. Additionally, as mentioned above, in some embodiments, the first type of memory or the second type of memory, or both, comprises groups of memory cells exhibiting different storage characteristics. For example, the first memory device 123 may have a first media type 124, and the second memory device 125 may have an associated second media type 126.

[0054] As mentioned above, in some embodiments, the first memory device or the second memory device has a higher bandwidth than the other of the first memory device or the second memory device, and the processing unit 122 can perform operations to reallocate processing resources available for the first memory device and the second memory device, such that a greater amount of processing resources are available for the memory device with higher bandwidth to perform operations to classify unknown objects. However, the embodiments are not limited thereto, and in some embodiments, the first memory device or the second memory device has a faster memory access time than the other of the first memory device or the second memory device, and the processing unit 122 can perform operations to reallocate processing resources available for the first memory device and the second memory device, such that a greater amount of processing resources are available for the memory device with faster memory access time to perform operations to classify unknown objects. Therefore, in at least one embodiment, the first memory device 123 is a NAND memory device including multiple single-level cells, a high-bandwidth memory device, or a DRAM device, and the second memory device 125 is a three-dimensional (3D) cross-point memory device or a NAND memory device including multiple multi-level cells.

[0055] In some embodiments, processing unit 122 may request information from a base station corresponding to at least one captured image associated with an unknown object. In such embodiments, the requested information includes confidence information about at least one pixel of an image stored by the base station, and / or that the image stored by the base station is determined to be similar to the unknown object. As used herein, the term "similar" generally refers to a condition where the characteristics of one object are sufficiently similar to the characteristics of another object such that the objects should be classified as the same object with a very high probability. Returning to the stop flag example above, an image without a smudged stop flag can be determined to be similar to a smudged stop flag when a threshold confidence parameter (which may be based on the confidence level of pixels in an image without a smudged stop flag) is met or exceeded.

[0056] Continuing with this non-limiting example, processing unit 122 may receive an indication that the device has entered a network coverage area generated by different base stations and, in response to receiving the indication, perform subsequent operations to reallocate computing resources between a first memory device and a second memory device. For example, based on determining that the device and therefore the autonomous vehicle has entered a network coverage area generated by different base stations, operations may be performed to reallocate or pre-allocate computing resources between the first and second memory devices, such that the computing resources are optimized to perform operations that might be necessary when the device operates within a network coverage area generated by different base stations.

[0057] However, the embodiments are not limited to this, and in some embodiments, in response to updated traffic sequence prediction modeling information determined or received by the device, subsequent operations may occur involving the reallocation (or pre-allocation) of computing resources between the first and second memory devices. For example, the device may determine or receive information corresponding to a change in traffic sequence prediction information being used by the device at a certain distance ahead of the autonomous vehicle. In response to this determination, computing resources may be reallocated (or pre-allocated) between the first and second memory devices such that the computing resources are optimized to perform operations that may be needed when the autonomous vehicle enters an area where the traffic sequence prediction information has changed.

[0058] An example from the above can occur when the device determines or receives information indicating that the updated traffic sequence prediction information corresponds to a situation where traffic volume on roads will be greater than previously expected. In this case, computing resources can be reallocated (or pre-allocated) between the first and second memory devices such that a larger amount of the fastest (e.g., highest bandwidth, lowest memory access time, etc.) computing resources (e.g., resources associated with the first memory device) are available before the autonomous vehicle encounters an area experiencing increased traffic. Conversely, if the device determines or receives information indicating that the updated traffic sequence prediction information corresponds to a situation where traffic on roads will be less than previously expected, then computing resources can be reallocated (or pre-allocated) between the first and second memory devices such that a smaller amount of the fastest (e.g., highest bandwidth, lowest memory access time, etc.) computing resources (e.g., resources associated with the first memory device) are available before the autonomous vehicle encounters an area experiencing decreased traffic.

[0059] Figure 1 Embodiments may include additional circuitry not described to avoid obscuring the embodiments of this disclosure. For example, memory system 104 may include address circuitry to latch address signals provided via I / O connections through I / O circuitry. Address signals may be received and decoded by row decoders and column decoders to access memory system 104 and / or memory devices 123, 125. Those skilled in the art will appreciate that the number of address input connections may depend on the density and architecture of memory system 104 and / or memory devices 123, 125.

[0060] Figure 2 This is another functional block diagram of a computing system 200 comprising a device including a host 202 and a memory system 204, according to several embodiments of the present disclosure. In some embodiments, the computing system 200 may be at least a portion of an electronic control unit (ECU) of an autonomous vehicle, such as those described herein. Figure 1The ECU 101 described herein. The memory system 204 may include several different memory devices 223, 225, 227, which may include one or more different media types 224, 226, 228. The different memory devices 223, 225, and / or 227 may include one or more memory modules (e.g., single in-line memory modules, dual in-line memory modules, etc.). The host 202, memory system 204, controller 220, processing unit 222, memory devices 223, 225, 227, and / or media types 224, 226, 228 may be similar to those described herein. Figure 1 The host 102, memory system 104, controller 120, processing unit 122, memory devices 123, 125 and / or media types 124, 126 described herein.

[0061] In some embodiments, each of memory devices 223, 225, and 227 may be a different type of memory device. Therefore, in some embodiments, each of memory devices 223, 225, and 227 may include different media types 224, 226, and 228. In a non-limiting example, memory device 223 may be a volatile memory device, such as a DRAM device, and may include media type 224 corresponding to a DRAM memory device (e.g., an array of memory cells including at least one capacitor and at least one transistor). Continuing this example, memory device 225 may be a flash memory device, such as a NAND memory device, and may include media type 226 corresponding to a NAND memory device (e.g., an array including floating gate metal-oxide-semiconductor field-effect transistors). In this non-limiting example, memory device 227 may be an emerging memory device (e.g., as described herein). Figure 4 The emerging memory device 439 described herein, such as the emerging memory device described above, may include a media type 228 corresponding to the emerging memory device (e.g., an array of resistive variable memory cells configured to perform bit storage based on changes in the volume resistance associated with the resistive variable memory cells).

[0062] Memory devices 223, 225, and 227 may be configured to read, write, and / or store data corresponding to one or more workloads executed by computing system 200 to detect and distinguish unknown objects detected by autonomous vehicles. For example, an application corresponding to the workload may be executed by processing unit 222 such that data written to memory devices 223, 225, and 227 is processed and analyzed by imaging devices (e.g., imaging devices). Figure 1 The imaging device 121 described herein is used as an operating autonomous vehicle (e.g., when capturing an unknown object). Figure 5 The autonomous transportation vehicles described in the document (542).

[0063] For example, if data corresponding to a specific workload is stored in memory device 223, then controller 220 and / or processing unit 222 may, in response to determining that the workload (e.g., a workload involving distinguishing unknown objects) can be performed more efficiently (e.g., optimized) using different memory devices, cause at least a portion of the data corresponding to the specific workload to be written to memory device 225 and / or memory device 227.

[0064] In such an example, processing unit 222 may determine the characteristics of the workload being performed when data is written to memory device 223, memory device 225, or memory device 227 by monitoring at least one of the access frequency of data associated with the workload, the latency associated with the execution of the workload, and / or the amount of processing resources consumed when the workload is executed, and write at least a portion of the data associated with the workload to at least one of the other two memory devices 223, memory device 225, or memory device 227 based at least in part on the determined access frequency associated with the workload, the latency associated with the execution of the workload, and / or the amount of processing resources consumed when the workload is executed.

[0065] Figure 3 This is a functional block diagram of a device including a memory system 304 according to several embodiments of the present disclosure. Figure 3 The memory system 304 is described below, and it may be similar to the one described herein. Figure 1 The memory system 104 and / or described herein Figure 2 The memory system 204 described herein. For example... Figure 3 As shown in the diagram, memory system 304 includes controller 320 (which may be similar to that described herein). Figure 1 The controller 120 and / or described herein Figure 2 The controller 220 described herein and the DRAM memory device 331 (which may be similar to those described herein) Figure 1 One or more of the memory devices 123 and 125 described herein Figure 2 The memory device 223, 225, 227 described herein) and the NAND memory device 333 (which may be similar to the one described herein) Figure 1 One or more of the memory devices 123 and 125 described herein Figure 2 (One of the memory devices 223, 225, 227 described herein). In some embodiments, the memory system 304 may reside on an electronic control unit, such as those described herein. Figure 1 ECU 101 as described in the document.

[0066] As in Figure 3As shown, the NAND memory device 333 may include various portions of memory cells, including a set of single-level memory cells (SLC) 335 and a set of multi-level memory cells (MLC), such as a set of three-level memory cells (TLC) 337. In some embodiments, the controller may cause data corresponding to an image of an unknown object to be written to the SLC portion 335 and / or the TLC portion 337 as part of an operation performed by an autonomous vehicle to identify the unknown object.

[0067] For example, data classified as hot data may be written to SLC portion 335, while data classified as cold data may be written to TLC portion 337, and vice versa, as part of optimizing the performance of memory system 304 during operations to distinguish unknown objects. By selectively writing portions of data corresponding to undistinguished objects to different memory portions of NAND memory device 333 (e.g., to SLC portion 335 and / or TLC portion 337), the performance of the computing system (especially during operations to distinguish unknown objects as described herein) can be improved compared to some methods. However, embodiments are not limited to this, and in some embodiments, hot data corresponding to unknown objects may be written to DRAM memory device 331, colder data corresponding to unknown objects may be written to NAND memory device 333, and cold data may be written to emerging memory devices (e.g., as described herein). Figure 4 (as illustrated in the description of emerging memory device 439). However, the embodiments are not limited thereto, and in some embodiments, data corresponding to unknown objects may be written to faster or higher bandwidth memory devices (e.g., the SLC portion 335 of NAND memory device 333 and / or DRAM memory device 331), while data corresponding to other operations performed by autonomous vehicles may be written to slower or lower bandwidth memory devices (e.g., the TLC portion 337 of NAND memory device 331 and / or emerging memory devices).

[0068] For example, by selectively writing portions of data corresponding to workloads that benefit from fast execution (e.g., performing operations to identify unknown objects) to DRAM memory device 331, while writing portions of data corresponding to workloads that may not benefit as much from fast execution (e.g., other operations that may not be as time-sensitive or critical as fast identification of unknown objects) to SLC portion 335 and / or TLC portion 337 and / or emerging memory devices (e.g., Figure 4 The emerging memory device 439 described herein can distribute workloads to memory devices within memory system 304 that allow for optimized execution of workloads within memory system 304. For similar reasons, portions of the workload can be written to emerging memory devices (e.g., those described herein). Figure 4 The emerging memory device 439 described herein).

[0069] Figure 4 This is another functional block diagram in the form of a device including a memory system 404 according to several embodiments of the present disclosure. Figure 4 The memory system 404 is described, and it may be similar to the one described herein. Figure 1 The memory system 104 described herein Figure 2 The memory system 204 and / or described herein Figure 3 The memory system 304 described herein.

[0070] As in Figure 4 As shown in the diagram, memory system 404 includes controller 420 (which may be similar to that described herein). Figure 1 The controller 120 described herein Figure 2 The controller 220 and / or described herein Figure 3 The controller 320 described herein and the DRAM memory device 431 (which may be similar to those described herein) Figure 1 One of the memory devices 123 and 125 described herein, Figure 2 One of the memory devices 223, 225, 227 and / or described herein Figure 3 One of the DRAM memory devices 331 described herein), and NAND memory device 433 (which may be similar to the one described herein). Figure 1 One of the memory devices 123 and 125 described herein, Figure 2 One of the memory devices 223, 225, 227 and / or described herein Figure 3 The NAND memory device 333 and emerging memory device 439 described herein (which may be similar to those described herein) Figure 1 One or more of the memory devices 123 and 125 described herein Figure 2 (One of the memory devices 223, 225, and 227 described herein).

[0071] DRAM memory device 431 may include an array of memory cells, the array including at least one transistor and a capacitor configured to store charge corresponding to a single data bit. NAND memory device 433 may include portions of memory cells, which may include a set of single-level memory cells (SLC) 435 and a set of multi-level memory cells (MLC), such as a set of three-level memory cells (TLC) 437, which may be similar to those described herein. Figure 3 The descriptions and explanations are provided in sections 335 (SLC) and 337 (TLC).

[0072] Emerging memory device 439 may be an emerging memory device as described above. For example, emerging memory device 439 may be a variable resistance (e.g., 3-D crosspoint (3D XP)) memory device, a memory device containing a self-select memory (SSM) cell array, or any combination thereof.

[0073] Figure 5 This is a schematic diagram of an autonomous vehicle 541 including an electronic control unit (ECU) 501 according to several embodiments of the present disclosure. (As shown in...) Figure 5 As shown in the diagram, the autonomous vehicle 541 communicates with the base station 543 via communication path 545. The ECU 501 can be similar to that described in this paper. Figure 1 ECU 101 as described herein. (As in...) Figure 5 As shown, the unknown object 547 may be located in the driving path of the autonomous vehicle 541.

[0074] Unknown object 547 may be an object or obstacle located along or adjacent to the driving path of autonomous vehicle 541. As mentioned above, unknown object 547 may be an object or obstacle that is not easily identified by autonomous vehicle 541 (e.g., by ECU 501 of autonomous vehicle 541) because unknown object 547 has not been previously observed by autonomous vehicle 541, is not recorded in the object or obstacle database accessible to autonomous vehicle 541, and / or has been obscured, defaced, or otherwise altered so that autonomous vehicle 541 cannot identify unknown object 547.

[0075] like Figure 5 As shown, imaging device 521 can receive information (e.g., images and / or videos) related to unknown object 547. For example, ECU 501 can be used to process and / or analyze said information within autonomous vehicle 541. In some embodiments, information (e.g., images and / or videos) may be processed by autonomous vehicle 541 as part of performing operations to identify unknown object 547 to determine what unknown object 547 is. As described above, performing operations to identify unknown object 547 can generate demanding workloads. Therefore, as described herein, information may be selectively written to different memory devices (e.g., as described herein) based on the characteristics of the workload. Figure 2 The memory devices 223, 225 and / or 227 described herein, and therefore different media types (e.g., those described herein) Figure 2 The media types described herein are 224, 226 and / or 228.

[0076] In a non-limiting example, the system may include an electronic control unit (ECU) 501 residing on the autonomous vehicle 541. (As described above...) Figure 1The ECU 501 is described as including a first memory device containing a first type of media (e.g., having...). Figure 1 The memory device 123 of media type 124 described herein), and the second memory device containing the second type of media (e.g., having Figure 1 The memory device 125 of the media type 126 described herein), and the imaging device (e.g., Figure 1 The imaging device 121 described herein and the processing unit coupled to the first memory device, the second memory device and the imaging device (e.g., Figure 1 The processing unit 122 described herein).

[0077] In some embodiments, the processing unit may determine that the first memory device exhibits better performance characteristics than the second memory device, and vice versa. As described above, the processing unit may also perform traffic sequence prediction operations to determine the number of objects that the autonomous vehicle will encounter within a threshold time period that is greater than a threshold object quantity. In response to the determination that the autonomous vehicle will encounter a number of objects that is greater than the threshold object quantity within the threshold time period, the processing unit may pre-allocate processing resources available for the autonomous vehicle from the second memory device to the first memory device.

[0078] The processing unit may then cause the imaging device to capture at least one image corresponding to an unknown object 547 positioned within the line of sight of the imaging device (e.g., an unknown object 547 positioned along the driving path of an autonomous vehicle). The processing unit may further determine that the first memory device exhibits better performance characteristics than the second memory device. In some embodiments, the processing unit may determine that the first memory device exhibits better performance characteristics by determining that the first memory device exhibits at least one or both of higher bandwidth or faster memory access time than the second memory device.

[0079] After processing resources available for autonomous vehicles are pre-allocated from the second memory device to the first memory device, the processing unit can use the pre-allocated processing resources to perform operations involving at least one captured image corresponding to an unknown object in order to classify the unknown object 547.

[0080] In some embodiments, the processing unit may transfer information stored in a first memory device to a second memory device in response to the capture of at least one image, thereby increasing the amount of available memory resources associated with the first memory device. This ensures that there are sufficient available memory resources in the faster memory device to store and process incoming images of the unknown object 547.

[0081] In some embodiments, the processing unit may receive information corresponding to an unknown object 547 from a base station 543 communicating with the autonomous vehicle 541. For example, the base station 543 may have previously received information corresponding to the unknown object 547 from other autonomous vehicles that encountered it and / or other autonomous vehicles that have communicated with the base station 543. In such embodiments, the processing unit may use reallocated processing resources to perform operations involving data corresponding to the unknown object and the received information corresponding to the unknown object 547 to classify the unknown object 547.

[0082] In some embodiments, when an image stored by a base station is determined to resemble an unknown object 547, the processing unit may receive confidence information about at least one pixel of the image stored by base station 543 as part of the information received from base station 543 corresponding to the unknown object 547. For example, if autonomous vehicle 541 and / or base station 543 determine that an image stored by base station 543 or autonomous vehicle 541 resembles an unknown object 547, then base station 543 or autonomous vehicle 541 may generate information corresponding to the confidence level of base station 543 or autonomous vehicle 541 that one or more pixels of the similar image correspond to the unknown object 547.

[0083] Continuing with the example above, the autonomous vehicle may further include an intelligent network capable of initiating operations involving at least one captured image corresponding to an unknown object 547 to classify the unknown object. As used herein, "intelligent network" generally refers to a network that contains sufficient intelligence to perform data identification and transmission by the network itself through protocols that automatically identify what things are (e.g., via deep learning) and can verify, confirm, and route transactions within the network.

[0084] Figure 6 This is a flowchart illustrating example methods corresponding to autonomous vehicle object detection according to several embodiments of the present disclosure. Method 650 can be executed by processing logic, which may include hardware (e.g., processing unit, processing device, control circuitry system, dedicated logic, programmable logic, microcode, device hardware and / or integrated circuits, etc.), software (e.g., instructions that run or execute on the processing unit), or a combination thereof. Although shown in a specific sequence or order, the order of processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be executed in different orders, and some processes may be executed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are possible.

[0085] At box 652, method 650 may include receiving an indication that the autonomous vehicle has entered a network coverage area generated by the base station. The autonomous vehicle may be similar to Figure 5 The autonomous vehicle 541 described herein, and the base station can be similar to Figure 5 The base station 543 is described in the document.

[0086] At block 654, method 650 may include performing an operation in response to receiving an instruction to reallocate (or pre-allocate) computing resources among multiple different types of memory devices associated with the autonomous vehicle. As described above, the memory devices may comprise different media types. In some embodiments, one of the memory devices may resemble memory devices 123, 223, while the different memory devices may resemble... Figure 1 and 2 The memory devices 125, 225, and / or 227 described herein. Therefore, a plurality of memory devices may comprise at least two different types of memory devices. Furthermore, the media type of one of the memory devices may be similar to media type 124, 224, while the media types of different memory devices may be similar to those described herein. Figure 1 and 2 The media types described in the document are 126, 226, and 228.

[0087] In some embodiments, method 650 may include: reallocating processing resources such that a memory device among a plurality of memory devices exhibiting higher bandwidth than another memory device among a plurality of memory devices can be used to receive at least one image; and using the memory device exhibiting higher bandwidth than another memory device among a plurality of memory devices to perform operations involving data corresponding to unknown objects to classify unknown objects. However, embodiments are not limited thereto, and in some embodiments, method 650 may include: reallocating processing resources such that a memory device among a plurality of memory devices exhibiting faster memory access time than another memory device among a plurality of memory devices can be used to receive at least one image; and using the memory device exhibiting faster memory access time than other memory devices among a plurality of memory devices to perform operations involving data corresponding to unknown objects to classify unknown objects. In some embodiments, the access time of the memory device may correspond to the type of interface utilized by the memory device. For example, a memory device communicatively coupled to the processing unit via a DDR4 or DDR5 interface may exhibit different access times than a memory device communicatively coupled to the processing unit via an NVMe interface.

[0088] At box 656, method 650 may include capturing data corresponding to an unknown object positioned within the line of sight of the autonomous vehicle. In some embodiments, the data corresponding to the unknown object may be captured by an imaging device, such as those described herein. Figure 1 The imaging device 121 described herein. In some embodiments, method 650 may include comparing at least one captured image corresponding to an unknown object with at least one image stored by the autonomous vehicle that has been determined to resemble the unknown object to classify the unknown object. In some embodiments, data may be captured by the imaging device while it is stationary on the autonomous vehicle.

[0089] At box 658, method 650 may include using reallocated computing resources to perform operations involving data corresponding to unknown objects to classify the unknown objects.

[0090] In some embodiments, method 650 may include receiving information corresponding to an unknown object from a base station and performing operations involving data corresponding to the unknown object and the received information corresponding to the unknown object using reallocated processing resources to classify the unknown object. In some embodiments, method 650 may further include receiving confidence information about at least one pixel of an image stored by the base station as part of receiving information corresponding to the unknown object from the base station, wherein the image stored by the base station is determined to resemble the unknown object.

[0091] Method 650 may further receive satellite imaging information as part of receiving information corresponding to an unknown object from a base station. In some embodiments, the satellite imaging information may include one or more satellite images or videos of an area near the autonomous vehicle where the unknown object is located. The autonomous vehicle may use the satellite imaging information as part of an operation to classify the unknown object. However, embodiments are not limited thereto, and in some embodiments, the satellite imaging may be directly received by the autonomous vehicle and may be used by the autonomous vehicle (e.g., processed by the autonomous vehicle's ECU) as part of an operation to classify the unknown object.

[0092] Method 650 may further include: performing operations, at least in part, based on received and / or determined traffic sequence prediction modeling information, to reallocate computational resources among multiple different types of memory devices associated with the autonomous vehicle. In some embodiments, the method may include: performing a traffic sequence prediction operation by the autonomous vehicle or a component thereof before capturing at least one image corresponding to an unknown object; and pre-allocating processing resources available for use with the multiple memory devices associated with the autonomous vehicle before capturing at least one image corresponding to an unknown object in response to the traffic sequence prediction operation indicating that the autonomous vehicle will encounter a number of objects greater than a threshold number within a threshold time period. In such embodiments, the method may further include using the pre-allocated processing resources to perform operations involving at least one image corresponding to an unknown object to classify the unknown object.

[0093] Method 650 may further include: receiving an indication from the autonomous vehicle that the autonomous vehicle has entered a network coverage area generated by different base stations; and / or performing subsequent operations in response to receiving the indication to reallocate computing resources among multiple different types of memory devices associated with the autonomous vehicle.

[0094] Figure 7 This is a schematic diagram illustrating a non-transitory computer-readable storage medium 760 according to several embodiments of the present disclosure. Although the machine-readable storage medium 624 is shown as a single medium in the exemplary embodiments, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for machine execution and causing the machine to perform any or more of the methods of the present disclosure. Therefore, the term "machine-readable storage medium" should be considered to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0095] like Figure 7 As shown, the computer-readable medium 760 includes instructions 762, which, when executed by the processing unit 722, cause the processing unit 722 to receive, via a radio frequency integrated circuit (RFIC) coupled to the processing unit, instructions corresponding to an autonomous vehicle entering a network coverage area generated by a base station.

[0096] The computer-readable medium 760 further includes instructions 764, which, when executed by the processing unit 722, cause the processing unit 722 to perform an operation in response to receiving an instruction to reallocate computing resources between a first memory device and a second memory device associated with the autonomous vehicle.

[0097] The computer-readable medium 760 further includes instructions 766, which, when executed by the processing unit 722, cause the processing unit 722 to cause an imaging device associated with the autonomous vehicle to capture at least one image corresponding to an unknown object positioned within the line of sight of the imaging device.

[0098] The computer-readable medium 760 further includes instructions 768, which, when executed by the processing unit 722, cause the processing unit 722 to perform operations involving at least one captured image corresponding to an unknown object, using reallocated computing resources, to classify the unknown object. In some embodiments, the computer-readable medium 722 may further include instructions, which, when executed by the processing unit 722, cause the processing unit 722 to perform operations to reallocate computing resources between a first memory device and a second memory device, such that the first memory device or the second memory device having associated higher bandwidth is reallocated for performing operations to classify the unknown object.

[0099] The computer-readable medium 760 may further include instructions that, when executed by the processing unit 722, cause the processing unit to receive information corresponding to an unknown object from a base station via RIFC and cause it to perform operations involving data corresponding to the unknown object and the received information corresponding to the unknown object using reallocated computing resources to classify the unknown object.

[0100] In some embodiments, the computer-readable medium 760 may further include instructions that, when executed by the processing unit 722, cause the processing unit to perform a traffic sequence prediction operation to determine traffic sequence prediction modeling information associated with the operating area of ​​the autonomous vehicle, and to perform an operation to reallocate computing resources between a first memory device and a second memory device, based at least in part on the determined traffic sequence prediction modeling information.

[0101] The computer-readable medium 760 may further include instructions that, when executed by the processing unit 722, cause the processing unit to request from the base station information corresponding to at least one captured image associated with an unknown object. In such embodiments, the requested information includes confidence information about at least one pixel of an image stored by the base station, and / or the image stored by the base station can be determined to resemble an unknown object.

[0102] In some embodiments, the computer-readable medium 760 may include instructions that, when executed by the processing unit 722, cause the processing unit 722 to receive via RFIC an indication that the autonomous vehicle has entered a network coverage area generated by different base stations, and in response to receiving the indication, perform subsequent operations to reallocate computing resources among multiple different types of memory devices associated with the autonomous vehicle.

[0103] Although specific embodiments have been described and illustrated herein, those skilled in the art will understand that arrangements calculated to achieve the same results may be substituted for the specific embodiments shown. This disclosure is intended to cover adaptations or variations of one or more embodiments of this disclosure. It should be understood that the foregoing description has been carried out in an illustrative rather than restrictive manner. Those skilled in the art will understand, upon reviewing the foregoing description, combinations of the foregoing embodiments and other embodiments not explicitly described herein. The scope of one or more embodiments of this disclosure includes other applications using the above-described structures and processes. Therefore, the scope of one or more embodiments of this disclosure should be determined with reference to the appended claims and the full scope of their equivalents.

[0104] In the foregoing detailed embodiments, for the purpose of simplifying this disclosure, some features are grouped in a single embodiment. This approach of the disclosure should not be construed as reflecting an intention that the disclosed embodiments of the disclosure must use more features than expressly stated in each claim. Rather, as reflected in the appended claims, the subject matter of the invention exists in fewer than all the features of a single disclosed embodiment. Therefore, the appended claims are hereby incorporated into the detailed embodiments, wherein each claim is an independent, separate embodiment.

Claims

1. A method for object detection in autonomous vehicles, comprising: Receive an indication that the autonomous vehicle (541) has entered the network coverage area generated by the base station (543); In response to receiving the instruction, an operation is performed to reallocate computing resources among multiple different types (124, 126, 228) of memory devices (123, 125, 227) associated with the autonomous vehicle (541); Data corresponding to an unknown object (547) placed within the line of sight of the autonomous vehicle (541) is captured; and The reallocated computing resources are used to perform operations involving the data corresponding to the unknown object (547) to classify the unknown object (547).

2. The method according to claim 1, further comprising: Receive information corresponding to the unknown object from the base station; and The reallocated computing resources are used to perform the operations involving the data corresponding to the unknown object and the received information corresponding to the unknown object in order to classify the unknown object.

3. The method of claim 2, further comprising receiving satellite imaging information as part of receiving information corresponding to the unknown object from the base station.

4. The method of claim 2, further comprising receiving confidence information about at least one pixel of an image stored by the base station as part of receiving information from the base station corresponding to the unknown object, wherein the image stored by the base station is determined to resemble the unknown object.

5. The method of claim 1, further comprising comparing the data corresponding to the unknown object with at least one image stored by the autonomous vehicle that is determined to resemble the unknown object.

6. The method of claim 1, further comprising performing the operation to reallocate the computing resources among the plurality of different types of memory devices associated with the autonomous vehicle, based at least in part on the received traffic sequence prediction modeling information.

7. The method of claim 1, further comprising: The autonomous vehicle receives an indication that it has entered a network coverage area generated by different base stations; and In response to receiving the instruction, subsequent operations are performed to reallocate computing resources among the plurality of different types of memory devices associated with the autonomous vehicle.

8. An apparatus for detecting objects in autonomous vehicles, comprising: Autonomous vehicles (541), which include: A first memory device (123, 223) includes a first type of media (124, 224); The second memory device (125, 225) includes a second type of media (126, 226); Imaging device (121); and Processing units (122, 222) coupled to the first memory device (123, 223), the second memory device (125, 225), and the imaging device (121), wherein the processing units (122, 222) are configured to: Receive an indication that the device has entered the network coverage area generated by the base station (543); In response to receiving the instruction, an operation is performed to reallocate computing resources between the first memory device (123, 223) and the second memory device (125, 225); This causes the imaging device (121) to capture at least one image corresponding to an unknown object (547) placed within the line of sight of the imaging device (121); and The reallocated computing resources are used to perform operations involving at least one captured image corresponding to the unknown object to classify the unknown object (547).

9. The apparatus of claim 8, wherein the processing unit is configured to: Perform traffic sequence prediction operations to determine traffic sequence prediction modeling information associated with the operating area of ​​the autonomous vehicle; and The operation is performed, at least in part, based on determined traffic sequence prediction modeling information, to reallocate the computing resources between the first and second memory devices.

10. The apparatus of claim 8, wherein the processing unit is configured to: Receive information corresponding to the unknown object from the base station; and Using the reallocated computing resources, the operation involving at least one captured image corresponding to the unknown object and received information corresponding to the unknown object is performed to classify the unknown object.

11. The device according to claim 8, wherein: The first memory device or the second memory device has a higher bandwidth than the other of the first memory device or the second memory device, and The processing unit is configured to perform the operation of reallocating computing resources between the first memory device and the second memory device, such that a larger-than-threshold amount of memory with higher bandwidth is available to perform the operation to classify the unknown objects.

12. The apparatus of claim 8, wherein the processing unit is configured to: The base station requests information corresponding to at least one captured image associated with the unknown object. The requested information includes confidence information about at least one pixel of an image stored by the base station, and The image stored by the base station is determined to resemble the unknown object.

13. The apparatus of claim 8, wherein the processing unit is configured to: The device receives an indication that it has entered a network coverage area generated by different base stations; and In response to receiving the instruction, subsequent operations are performed to reallocate computing resources between the first memory device and the second memory device.

14. The device of claim 8, further comprising a radio frequency integrated circuit (RFIC) (111) residing on the autonomous vehicle, wherein the RFIC is used to facilitate communication between the autonomous vehicle and the base station.

15. A non-transitory computer-readable storage medium for autonomous vehicle object detection, comprising instructions that, when executed by a processing unit (122, 222), cause the processing unit (122, 222) to: An instruction is received via a radio frequency integrated circuit (111) that can be coupled to the processing unit (122, 222), the instruction corresponding to the autonomous vehicle (541) entering the network coverage area generated by the base station (543); In response to receiving the instruction, an operation is performed to reallocate computing resources between a first memory device (123, 223) and a second memory device (125, 225) associated with the autonomous vehicle (541); This causes the imaging device (121) associated with the autonomous vehicle (541) to capture at least one image corresponding to an unknown object (547) positioned within the line of sight of the imaging device (121); and This causes the use of reallocated computing resources to perform operations involving at least one captured image corresponding to the unknown object (547) to classify the unknown object (547).

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions are executable by the processing unit to cause the processing unit to: Receive information corresponding to the unknown object from the base station via the RIFC; and This causes the operation involving at least one captured image corresponding to the unknown object and received information corresponding to the unknown object to be performed using the reallocated computing resources in order to classify the unknown object.

17. The non-transitory computer-readable storage medium of claim 15, wherein the instructions are executable by the processing unit to cause the processing unit to: Perform traffic sequence prediction operations to determine traffic sequence prediction modeling information associated with the operating area of ​​the autonomous vehicle; and The operation is performed, at least in part, based on determined traffic sequence prediction modeling information, to reallocate the computing resources between the first and second memory devices.

18. The non-transitory computer-readable storage medium of claim 15, wherein the instructions are executable by the processing unit to cause the processing unit to perform the operation to reallocate computing resources between the first memory device and the second memory device, such that the first memory device or the second memory device having associated higher bandwidth is reallocated to perform the operation to classify the unknown object.

19. The non-transitory computer-readable storage medium of claim 15, wherein the instructions are executable by the processing unit to cause the processing unit to: The base station requests information corresponding to at least one captured image associated with the unknown object, wherein: The requested information includes confidence information about at least one pixel of an image stored by the base station, and The image stored by the base station was determined to resemble the unknown object.

20. The non-transitory computer-readable storage medium of claim 15, wherein the instructions are executable by the processing unit to cause the processing unit to: The autonomous vehicle receives an indication via the RFIC that it has entered a network coverage area generated by different base stations; and In response to receiving the instruction, subsequent operations are performed to reallocate computing resources between the first and second memory devices associated with the autonomous vehicle.

Images