Virtual mirror with automatic zoom based on vehicle sensors

Abstract

The present application relates to a virtual mirror with automatic zoom based on vehicle sensors. In one approach, a method includes: displaying image data obtained using a first field of view of a camera of a first vehicle to a user of a first vehicle, wherein the camera collects the image data of an object located outside the first vehicle; detecting a second vehicle by at least one processing device of the first vehicle; determining by the one processing device whether the second vehicle is within a predetermined area relative to the first vehicle; and in response to determining that the second vehicle is within the predetermined area, displaying image data obtained using a second field of view of the camera.

Description

Technical Field

[0001] At least some embodiments disclosed herein generally relate to collecting image data using a camera and more particularly, but not limited to, to collecting image data by a vehicle using a camera having a field of view that is adjusted based on detection of nearby vehicles. Background Art

[0002] Driver-controlled and autonomous vehicles may include many sensors to assist in controlling the vehicle. One type of sensor is a camera that provides image data. In some cases, the vehicle may use information about the environment collected by the vehicle's camera and other sensors to navigate. In other cases, the driver of the vehicle may use the collected information to control the operation of the vehicle. In addition to one or more cameras, a driver-controlled or autonomous vehicle may also include sensors such as lidar and / or radar.

[0003] When a driver controls the navigation of a vehicle, one of the challenges the driver faces is determining the presence of other vehicles operating around the driver's vehicle. In many cases, it is difficult for the driver to see other vehicles due to various blind spots associated with the vehicle. For example, certain configurations of rearview mirrors and side mirrors do not allow the driver to have full 360-degree visibility of other vehicles.

[0004] Generally speaking, a blind spot of a vehicle is an area around the vehicle that the driver cannot directly observe while controlling the vehicle. Blind spots can exist in various vehicles, such as airplanes, cars, motorboats, and trucks. In some cases, a blind spot occurs when the side view mirrors do not provide the driver with full visibility to the sides of the vehicle.

[0005] Currently, many vehicles employ advanced driver assistance systems to assist the driver. Advanced driver assistance systems (ADAS) are systems that provide feedback or automatically control a vehicle in response to defined inputs. One example of ADAS is a blind spot detection system. Another example is a system that maintains a safe following distance of a vehicle relative to other vehicles. Both systems encounter significant technical problems.

[0006] Current blind spot ADAS are installed in non-autonomous vehicles and are mandatory in autonomous vehicles. These systems currently rely on sensor-based technologies, such as by using lidar, radar or sonar to detect the presence of vehicles in the blind spot to detect vehicles within a certain distance of the driver's vehicle. While these systems may work under ideal conditions, they are significantly impaired in adverse weather conditions and, in general, are affected by environmental factors that may lead to false positives and other negative detection conditions. Summary of the invention

[0007] On the one hand, the present application provides a method comprising: displaying image data obtained using a first field of view of a camera of a first vehicle to a user of a first vehicle, wherein the camera collects the image data of an object located outside the first vehicle; detecting a second vehicle by at least one processing device of the first vehicle; determining by the one processing device whether the second vehicle is within a predetermined area relative to the first vehicle; and in response to determining that the second vehicle is within the predetermined area, displaying image data obtained using a second field of view of the camera.

[0008] On the other hand, the present application provides a system comprising: a camera mounted on the side of a first vehicle and configured to collect image data about objects outside the first vehicle; a display screen positioned for viewing by a driver of the first vehicle; at least one processing device; and a memory containing instructions configured to instruct the at least one processing device to: present an image on the display screen based on data collected using a first field of view of the camera; detect a second vehicle; determine whether the second vehicle is within a predetermined area; and in response to determining that the second vehicle is within the predetermined area, present an image on the display screen based on data collected using a second field of view of the camera.

[0009] On the other hand, the present application provides a non-transitory computer storage medium storing instructions that, when executed on at least one processing device, cause the at least one processing device to at least: display image data obtained using a first field of view of a camera of a first vehicle to a user of a first vehicle; detect a second vehicle; determine whether the second vehicle is within a predetermined area relative to the first vehicle; and in response to determining that the second vehicle is within the predetermined area, display image data obtained using a second field of view of the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0011] Figure 1 A camera mounted on the side of a vehicle to provide a rear view and / or a side view to the driver of the vehicle is shown according to one embodiment.

[0012] Figure 2 An example of a vehicle displaying image data from one or more cameras is shown according to one embodiment.

[0013] Figure 3An example computing system of a vehicle for processing data from sensors and providing a display to a driver is presented according to one embodiment.

[0014] Figure 4 An autonomous vehicle including a sensor module providing data to a central processing device is shown according to one embodiment.

[0015] Figure 5 A sensor module including a sensor providing raw data to a processor is shown according to one embodiment.

[0016] Figure 6 An approaching vehicle is shown overtaking a driver's vehicle according to one embodiment.

[0017] Figure 7 According to one exemplary embodiment, an oncoming vehicle is shown that is in a different lane from the driver's vehicle and is about to pass.

[0018] Figure 8 A vehicle is shown approaching from behind a driver's vehicle and then changing lanes in preparation for passing, according to one embodiment.

[0019] Fig. 9 A block diagram is shown of a computing device (eg, a server) that may be used in various embodiments.

[0020] Fig.10 A block diagram of a computing device (eg, a user's mobile device) is shown according to one embodiment.

[0021] Fig.11 A method for adjusting the field of view of a vehicle camera based on detection of nearby vehicles is presented according to one embodiment.

[0022] Fig.12 is a block diagram of a vehicle according to some embodiments. DETAILED DESCRIPTION

[0023] At least some embodiments herein relate to collecting data using a field of view that automatically adjusts when a nearby vehicle is detected by one or more cameras. In some embodiments, the use of the camera provides a "virtual" side mirror to provide the driver with a view of objects to the side of the vehicle operated by the driver. In other embodiments, due to the automatic adjustment of the field of view of the vehicle's camera or other sensor, the image data from the camera is used to provide improved navigation of the vehicle based on the improved degree of sensor coverage of the area around the vehicle.

[0024] Various embodiments related to cameras with adjustable fields of view as disclosed herein relate to systems and methods that may be used for driver-controlled or autonomous vehicles (e.g., to assist and / or control navigation based on image data collected from one or more cameras of the vehicle). The vehicle may be, for example, any of several types of vehicles (e.g., cars, trucks, airplanes, drones, watercraft, etc.). Implementations may include vehicles controlled by a driver, autonomous vehicles, fixed structures including devices that perform image processing and / or object detection (e.g., safety systems), and standalone camera sensor devices or systems.

[0025] Existing side-view mirrors on cars and other vehicles are typically mechanical structures in which mirrors are mounted to reflect light from physical objects so that the driver can see objects such as nearby vehicles in various areas around the vehicle. However, it is expected that in the future such existing mirrors will be replaced by virtual mirrors consisting of cameras mounted on the vehicle to provide a view outside the vehicle. In one example, the mounted camera provides a live video feed for presentation to the driver on a display such as an LCD screen inside the vehicle.

[0026] Although such virtual mirrors can reduce wind noise due to a small camera footprint and can accommodate intelligent safety features, such virtual mirrors still present technical issues of blind spots, where the driver cannot see certain areas around the vehicle. Some virtual mirrors can provide the driver with the ability to adjust the field of view of the virtual mirror based on personal preferences. However, in many cases, the driver's field of view adjustment still cannot remove all blind spots around the vehicle. The presence of these blind spots can create technical issues of increased probability of collision with another vehicle, which causes physical damage and / or injury.

[0027] Various embodiments of the present invention provide technical solutions to one or more of the above-mentioned technical problems. In some embodiments, a method is provided for automatically changing the field of view of a camera used as a virtual mirror based on the detection of a nearby vehicle. In one embodiment, a method includes: displaying image data obtained using a first field of view of a camera of a first vehicle (e.g., as used in a driver's virtual side-view mirror) to a user of a first vehicle (e.g., a driver), wherein the camera collects image data of an object located outside the first vehicle; detecting a second vehicle by at least one processing device of the first vehicle; determining by the one processing device whether the second vehicle is within a predetermined area relative to the first vehicle; and displaying image data obtained using a second field of view of the camera in response to determining that the second vehicle is within the predetermined area.

[0028] In one embodiment, the camera's field of view is automatically changed based on the detection of nearby vehicles by zooming the camera's lens in and out. This automatic zoom increases the safety of vehicle operation by providing the driver with a wider field of view that covers blind spots (e.g., blind spots where the driver's field of view is obstructed) when nearby vehicles or another vehicle is about to overtake the driver's vehicle.

[0029] Automatic zoom further increases safety by providing the driver with a narrower field of view when nearby vehicles are not detected to the side of the driver's vehicle on which the camera is mounted. By zooming in, the driver can be provided with a view of farther distances. One advantage is that the driver may have additional reaction time to respond to nearby vehicles.

[0030] In one embodiment, the maximum field of view of the camera when fully zoomed out is large enough to provide viewing coverage of the entire side of the vehicle and viewing coverage toward the rear of the vehicle. In one example, the maximum field of view is at least 110 degrees. When the camera's field of view is zoomed in, increased details of nearby objects such as the vehicle are provided to the driver. In one example, the field of view is no greater than, for example, 60 degrees when narrowed.

[0031] In some embodiments, the field of view of the camera for the virtual mirror is adjusted based at least in part on the speed of the vehicle. In one embodiment, when it is determined that the speed of the driver's vehicle is less than a predetermined speed, the field of view is adjusted to wide by pushing away. In one example, the vehicle speed is less than 50km / h. In one example, this wide field of view is used to provide the driver with a live video display covering the side of the vehicle (including any blind spots on the side removed), and also covers observation towards the rear of the vehicle.

[0032] In one embodiment, when it is determined that the speed of the driver's vehicle is greater than a predetermined speed, the camera is configured to have a narrow field of view (e.g., by zooming in on the camera's lens). In one example, the speed of the vehicle is determined to be greater than 50 km / h. Using this narrow field of view, the primary focus of the camera is on an area further away and toward the rear of the vehicle and includes only a portion of the side area of ​​the vehicle. In one example, this narrow field of view is comparable to the field of view typically provided by existing mechanical side mirrors.

[0033] In one embodiment, in response to determining that the speed of the vehicle is greater than a predetermined speed, the system further requires that there are no nearby vehicles on the side of the vehicle on which the camera is installed. If it is determined that there are nearby vehicles on the side of the vehicle, the camera will remain within the wide field of view even if the speed of the vehicle is greater than the predetermined speed.

[0034] In one embodiment, a computer system detects the presence of a nearby vehicle behind the driver's vehicle. The system also determines that the nearby vehicle is about to overtake the driver's vehicle. In one example, the system determines that the nearby vehicle will reach the side of the driver's vehicle in less than a predetermined time. In one example, the predetermined time is a time between 5 seconds and 30 seconds.

[0035] In response to determining both the presence of a nearby vehicle behind the driver's vehicle and that the nearby vehicle will overtake the driver's vehicle, the system causes the camera to change to a wide field of view. This allows the driver to view the sides of the vehicle and eliminates blind spots so that the driver can visually follow the nearby vehicle as it passes.

[0036] In one embodiment, the system determines that the nearby vehicle will overtake the driver's vehicle based on detecting that the nearby vehicle is within a predetermined distance behind the driver's vehicle. In one example, the predetermined distance is 30 meters or less. In one embodiment, the system determines that the nearby vehicle will overtake the driver's vehicle based on determining that the nearby vehicle has a speed greater than the speed of the driver's vehicle.

[0037] In one embodiment, the computer system detects nearby vehicles on the side of the driver's vehicle on which the camera is mounted. In one example, nearby vehicles are detected on the side even if no prior detection of nearby vehicles behind the driver's vehicle is performed. In response to detecting nearby vehicles on the side, the camera is changed from a narrow field of view to a wide field of view.

[0038] In one embodiment, after a nearby vehicle passes the driver's vehicle, the camera's field of view switches from the wider field of view back to the narrower field of view. In one example, the new narrower field of view has an angle that is different from the camera's original narrow field of view. In one example, the new narrower field of view is determined based on the context of the driver's vehicle. In one example, a neural network or other computer model is used to determine the new narrower field of view.

[0039] In one embodiment, after the nearby vehicle passes the driver's vehicle, before switching the camera back to the narrower field of view, the computer system further checks to determine that no other nearby vehicles are detected behind the car and / or to the side of the vehicle on which the camera is mounted. If no such vehicle is detected, the camera switches to the narrower field of view. In one example, when the nearby vehicle has passed the front occupancy area of ​​the driver's vehicle, it is determined that the nearby vehicle has passed the driver's vehicle.

[0040] In one embodiment, one or more sensors may be used to detect nearby vehicles. In one example, the sensors are radar sensors, ultrasonic sensors, and / or image sensors. In one embodiment, an artificial intelligence model is used to analyze data from these sensors. In one example, an object detection algorithm is applied to the data from these sensors.

[0041] In one embodiment, nearby vehicles may be detected based on one or more communications received by the driver's vehicle from nearby vehicles. In one example, the communications are V2X signals received from surrounding vehicles. In one example, the communications contain information about the location of the nearby vehicles. In one example, the driver's vehicle compares the location of the nearby vehicles to the location of the driver's vehicle. In one example, the nearby vehicles are detected based on the location of the nearby vehicles being less than a predetermined distance from the location of the driver's vehicle.

[0042] In one embodiment, the potential set of nearby vehicles to be detected may be limited. In one example, the set of nearby vehicles is limited to those vehicles determined to be moving in the same traffic direction as the driver's vehicle. In one example, the same traffic direction is determined based on the direction of the nearby vehicles being within a predetermined number of degrees relative to the direction of the driver's vehicle. In one example, the direction of travel of the nearby vehicles is within 45 degrees of the direction of travel of the driver's vehicle.

[0043] In another example, the set of nearby vehicles is limited to those vehicles that are within a predetermined distance from the driver's vehicle. In one example, the nearby vehicles are within a predetermined distance behind the driver's vehicle. In one example, the nearby vehicles are within a predetermined distance of the location of a camera for detecting nearby vehicles and / or for presenting images of nearby vehicles to the driver.

[0044] In another example, the group of nearby vehicles is limited to those vehicles within a predetermined viewing angle relative to the position on the driver's vehicle. In one example, the predetermined viewing angle is the currently used field of view of the camera (e.g., when used to detect nearby vehicles).

[0045] In one embodiment, multiple sensors are used to perform the detection of nearby vehicles by the driver's vehicle. One of these sensors may be a camera that provides image data of an image frame of the sensor used in the camera. When more than one sensor is used, a sensor fusion system may perform processing on the data inputs from the various sensors.

[0046] In some cases, artificial intelligence (AI) based sensor fusion is used. AI sensor fusion consists of sensors (e.g., image, radar, lidar sensors) and sensor fusion blocks (e.g., circuits or processing devices that execute image processing algorithms) that detect objects expected in the image (e.g., nearby vehicles). The sensor fusion algorithm(s) may be implemented on a dedicated processing device (e.g., SOC, FPGA, CPU, GPU, etc.).

[0047] In one embodiment, the sensor is a camera. The scanning window is incremented in the image frames of data collected by the sensor. In other embodiments, other types of sensors (e.g., lidar or radar sensors) may be used. In one embodiment, the scanning of the image frames of the sensor is performed using the scanning window with known existing image processing algorithms to detect objects within the scanning window.

[0048] Figure 1 Cameras 150, 152 are shown mounted on the side of a vehicle 7111 to provide a rear view and / or side view to the driver of the vehicle 7111 according to one embodiment. Camera 150 has a narrower field of view with an angle 154. Camera 152 has a wider field of view with an angle 156. Camera 150 can be used to display images of nearby objects around the vehicle 7111 to the driver (e.g., using an LCD screen, a head-up display projected on a windshield, and / or other types of display devices). In one example, camera 150 provides a live video stream including an object 7157 displayed to the driver. In one example, object 7157 has been detected within a predetermined zone 164. In one example, predetermined zone 164 is defined at least in part based on a predetermined distance (e.g., a distance range between 20 meters and 60 meters) behind the vehicle 7111.

[0049] Blind spot area 160 is an area that is blind to the driver. In one example, object 7157 is a nearby vehicle that has been detected by vehicle 7111. In response to determining that the nearby vehicle is overtaking vehicle 7111, the field of view of camera 150 is changed to have a wider angle so that the driver can observe the area around the side of vehicle 7111, which includes blind spot area 160. This allows the driver to observe object 7157 as it passes vehicle 7111. Once object 7157 passes in front of vehicle 7111, the field of view of camera 150 returns to angle 154.

[0050] In one example, the computer system of vehicle 7111 has detected object 7155 to the side of vehicle 7111 in predetermined side zone 162. In response to this detection, camera 152 is switched by the computer system to have a wider field of view with angle 156.

[0051] In one embodiment, the vehicle 7111 has a wireless connection 7115 for communicating with other vehicles. In one example, the wireless connection 7115 includes a transceiver that sends and receives signals, such as to send or receive signals to other vehicles around the vehicle 7111. The vehicle 7111 can use these signals to determine that another vehicle is nearby.

[0052] In one embodiment, event data is extracted from the non-volatile memory of the vehicle 7111 and communicated to the central server via a wireless communication network. The central server analyzes the event data (e.g., to further process the image data to support detection of objects by the vehicle 7111). In one example, the server selects the type of event data to be downloaded for analysis (e.g., sensor data or control system status). An example of a wireless communication network is a cellular telephone network.

[0053] In response to determining or identifying detection of the object (e.g., by the server and / or the vehicle 7111), at least one action is performed. For example, the server may transmit a communication to the vehicle 7111 that causes the vehicle to reconfigure software (e.g., change a configuration setting of a neural network that performs image analysis, or reconfigure software that controls the field of view of the cameras 150, 152).

[0054] In one embodiment, vehicle 7111 may receive a signal from a central server that is used by vehicle 7111 to identify that another vehicle is nearby. The signal may cause vehicle 7111 to switch the field of view of camera 150 and / or 152 (e.g., to prepare for another vehicle to pass vehicle 7111).

[0055] In one example, a vehicle may communicate with a central server via wireless connection 7115 to submit event data (e.g., location and / or identity / type of nearby vehicles). Wireless connection 7115 may use a wireless local area network, a cellular communication network, and / or a communication link 7107 to a satellite 7109.

[0056] Figure 2 An example of a vehicle 7111 displaying image data from one or more cameras 170 is shown in accordance with one embodiment. In one example, the cameras 170 are Figure 1The camera 150, 152 of the vehicle 7111 is provided with the image data from the camera 170. The image data from the camera 170 is presented to the driver of the vehicle 7111 on the display screen 172. In one example, the image data is a live video stream (e.g., to show nearby vehicles around the vehicle 7111). In one example, the image data is augmented reality data superimposed on the live video stream. The augmented reality data is derived from an analysis of the raw image data obtained from the camera 170. In one example, an artificial neural network (ANN) model 119 is used to analyze the raw image data.

[0057] In one embodiment, the vehicle 7111 includes a sensor module 137 and is configured using the ANN model 119. The sensor module 137 provides object data to the computer 131. The computer 131 includes a processor and / or software process that acts as a central processing device.

[0058] In one example, the object data is collected data corresponding to an image captured by one of the cameras 170. In one example, the collected data may be processed by the sensor module to make an initial vehicle or other object detection decision to be confirmed or refuted by the computer 131. In one example, the other object is a person or an animal.

[0059] The vehicle 7111 includes an infotainment system 149, a communication device 139, and one or more sensor modules 137. The computer 131 is connected to some controls of the vehicle 7111, such as a steering control 141 for the direction of the vehicle 7111, a brake control 143 for stopping the vehicle 7111, an acceleration control 145 for the speed of the vehicle 7111, etc. In one example, the communication device 139 is used to communicate with other vehicles (e.g., nearby vehicles that send signals including the vehicle's location).

[0060] The computer 131 of the vehicle 7111 includes one or more processors 133 , a memory 135 storing firmware (or software) 127 , an ANN model 119 , and other data 129 .

[0061] In one example, firmware 127 is updated via wireless updates in response to a communication from the central server sent in response to an object detection (e.g., an initial object detection decision received from vehicle 7111 and confirmed by the central server after processing an initial object detection by vehicle 7111). Alternatively and / or in addition, other firmware of various computing devices or systems of vehicle 7111 may be updated.

[0062] The one or more sensor modules 137 may include visible light cameras, infrared cameras, lidar, radar or sonar systems and / or peripheral sensors configured to provide sensor input to the computer 131. The modules of firmware (or software) 127 executed in the processor(s) 133 apply the sensor input to the ANN defined by the model 119 to generate outputs, such as images or video clips, that recognize or classify objects captured in the sensor input. Data from this recognition and / or classification may be included in data collected by a memory device (e.g., a non-volatile memory device) and sent from the vehicle 7111 to a central server above.

[0063] Alternatively and / or in addition, the autonomous driving module of firmware (or software) 127 may use object detection (e.g., by a server and / or vehicle) to generate a response. The response may be a command to activate and / or adjust the field of view of one or more of the cameras 170 and / or one of the vehicle controls (e.g., 141, 143, and 145). In one embodiment, the response is an action performed by the vehicle, wherein the action has been configured based on an update command from a central server (e.g., the update command may be generated by the central server in response to object detection based on analysis of event data from the vehicle). In one embodiment, the vehicle is configured before generating a control response. In one embodiment, the configuration of the vehicle is performed by updating the firmware of the vehicle 7111. In one embodiment, the configuration of the vehicle includes updating a computer model (e.g., ANN model 119) stored in the vehicle 7111.

[0064] In one embodiment, the central server stores the received sensor input as part of the sensor data for subsequent further training or updating of the ANN model 119 using a supervised training module. When an updated version of the ANN model 119 is available to the central server, the vehicle 7111 may use the communication device 139 to download the updated ANN model 119 to be installed in the memory 135 and / or to replace the previously installed ANN model 119. These actions may be performed in response to determining that the vehicle 7111 failed to detect some nearby passing vehicles or other objects.

[0065] In one example, the output of the ANN model 119 may be used to control (e.g., 141, 143, 145) the acceleration of a vehicle (e.g., 7111), the speed of the vehicle 111, and / or the direction of the vehicle 7111 during autonomous driving. In one example, the output of the ANN model 119 is used to control the field of view of one or more of the cameras 170.

[0066] In one example, data obtained from a sensor of a vehicle (e.g., data for detecting nearby vehicles) may be images of objects captured using a camera that uses light visible to the human eye for imaging, or a camera that uses infrared light for imaging, or a sonar, radar, or lidar system. In one embodiment, image data obtained from at least one sensor of a vehicle is the analyzed portion of the data collected from the vehicle. In some cases, the ANN model is configured for a specific vehicle based on the sensor and other collected data.

[0067] Figure 3 An example computing system of a vehicle 101 is shown for processing data from sensors 103 and providing an image to a driver on a display 302, according to one embodiment. The computing system of the vehicle 101 processes the data from the sensors 103. In one embodiment, the collected data is obtained from image frames 105. The data collected from the sensors 103 is used, for example, to navigate the vehicle 101 and / or control the field of view of one or more cameras (e.g., cameras 150, 152) of the vehicle 101.

[0068] The data collected by sensor 103 within image frame 105 is provided to processor 107. When a new image within image frame 105 is collected by sensor 103, processor 107 executes an algorithm for object detection in the new image.

[0069] The processor 107 scans the collected data to detect one or more objects. As each object is detected, the data corresponding to the detected object is stored in a database of detected objects 113 in the memory 109. The image processing algorithm used to scan the new image can be stored in the memory 109 as a computer model 115. In one example, the computer model 115 includes a neural network and / or other machine learning algorithm.

[0070] In one embodiment, object detection includes training a computer model using at least one of supervised learning or unsupervised learning, and using the computer model to analyze video data. In one example, the computer model includes a neural network. In one example, the computer model is computer model 115. In one example, video data is collected by sensor 103.

[0071] In one embodiment, object detection using a scanning window within image frame 105 may include using machine learning techniques (e.g., a computer model such as training a neural network) to detect objects identified in video data collected by a camera.

[0072] In one embodiment, object detection may further include inputting video features extracted from the video data into a computer model. For example, video data for training the computer model may be collected before analyzing the above video data. In one example, the computer model may be one or more artificial neural networks. For example, the neural network may be trained based on video data collected by the sensor device.

[0073] In one embodiment, object detection may further include determining output from a computer model based on the extracted video data features. For example, the output from the computer model may be based on an output adjusted using the environment determined from the collected video data.

[0074] In one embodiment, processor 107 may include functionality for local processing of information without the need for a server or back-end computing device. In some embodiments, processor 107 includes an audio analysis module, a video analysis module and configuration settings, and other dedicated software units. Each of these modules may be stored as instructions and / or data on a memory device, such as memory 109 (e.g., EEPROM and / or other non-volatile memory). The specific configuration and software used to instantiate these modules may be modifiable and updateable (e.g., updateable from a remote computing device over a network).

[0075] Configuration settings(es) may also represent specific configurations, parameters, weights, or other settings of a quantitative model, classifier, machine learning algorithm, etc. As an example, a support vector machine (SVM) may be represented as a hyperplane that partitions two regions of a vector space into two corresponding classes. The coefficients defining the hyperplane may be included in the configuration settings(es). As another example, an artificial neural network (ANN) may include a set of interconnected nodes with specific weights between each node connection. These connection weights may also be included in the configuration settings(es).

[0076] The processor 107 may implement a method for detecting a specific audio event or a specific object within a video frame (e.g., a video frame within the image frame 105). In some embodiments, the processor 107 and / or other computing device includes a classifier or machine learning algorithm that is executed locally in the vehicle 101. The parameters or weights of the classifier or machine learning algorithm - i.e., (several) configuration settings - may be updated (e.g., received from the computing device via the communication module). Thus, the (several) configuration settings may include parameters, coefficients, or weights that enable the audio analysis module and the video analysis module to perform a specific task or improve the audio analysis module and the video analysis module to perform a specific task.

[0077] In some cases, various embodiments of computing systems including processor 107 and / or memory 109 may include instructions thereon that, when executed, implement computer vision and / or image or video analysis functions. Alternatively or in addition, a computing device (e.g., a device including processor 107) may include one or more integrated circuits that communicate with the processor(s) to perform aspects of computer vision functions, such as object recognition, image or video compression, and / or facial detection, among other functions.

[0078] As described herein, computer vision includes, for example, methods for acquiring, processing, analyzing, and understanding images and, in general, high-dimensional data from the real world in order to generate, for example, digital or symbolic information in the form of decisions. Image data can be in many forms, such as video sequences, views from multiple cameras, or multi-dimensional data from a scanner.

[0079] One aspect of computer vision includes determining whether image data contains a particular object, feature, or activity. Different kinds of computer vision recognition include: Object recognition (also called object classification) - can recognize one or several pre-specified or learned objects or object categories, usually together with their 2D position in the image or 3D pose in the scene. Recognition - recognize individual instances of objects. Examples include recognition of specific objects, recognition of landscapes or buildings, and / or recognition of specific types of vehicles. Detection - scan image data for specific conditions. Examples include detecting vehicles in automatic road toll collection systems. Detection based on relatively simple and fast calculations is sometimes used to find small areas of interesting image data that can be further analyzed by more computationally demanding techniques to produce a correct interpretation.

[0080] Typical functions and components (e.g., hardware) found in many computer vision systems are described in the following paragraphs. Embodiments of the present invention may include at least some of these aspects. For example, embodiments of vehicle 101 may include a computer vision module. The computer vision module may include any of the components (e.g., hardware) and / or functions described herein with respect to computer vision, including but not limited to one or more cameras, sensors, and / or processors. In some embodiments of the present invention, a microphone, camera, and / or imaging processor may be a component of the computer vision module.

[0081] Image acquisition - digital images are produced by one or several image sensors, which may include, in addition to various types of light-sensitive cameras, distance sensors, tomography devices, radar, ultrasound cameras, etc. Depending on the type of sensor, the resulting image data may be a 2D image, a 3D volume, or a sequence of images. The pixel values ​​may correspond to light intensity in one or several spectral bands (gray or color images), but may also be related to various physical measurements, such as depth, absorbance, or reflectance of acoustic or electromagnetic waves or nuclear magnetic resonance.

[0082] Preprocessing - Before applying a computer vision method to image data in order to extract some particular piece of information, it is often advantageous to process the data in order to ensure that it satisfies certain assumptions implicit in the method. Examples of preprocessing include, but are not limited to, resampling to ensure that the image coordinate system is correct, noise reduction to ensure that sensor noise does not introduce spurious information, contrast enhancement to ensure that relevant information can be detected, and scale-space representation to enhance image structure at local appropriate scales.

[0083] Feature Extraction - Extracting image features of varying levels of complexity from image data. Typical examples of such features are: lines, edges, and ridges; local points of interest such as corners, spots, or dots; more complex features may be related to texture, shape, or motion.

[0084] Detection / Segmentation - At some point in the processing, a decision can be made about which image points or regions of the image are relevant for further processing. Examples are: selecting a particular set of points of interest; segmenting one or more image regions containing objects of interest; segmenting an image into a nested scene structure consisting of foreground, object groups, single objects, or salient object parts (also known as a spatial classification scene hierarchy).

[0085] High-level processing - At this step, the input may be a small data set, such as a set of points or image regions assumed to contain a specific object. The remaining processing may include, for example: verifying that the data meets the assumptions based on the model and the specific application; estimating application-specific parameters such as object pose or object size; image recognition - classifying the detected objects into different categories; image registration - comparing and combining two different views of the same object.

[0086] Make Decisions—Make the final decision required for the application, such as identifying a match / mismatch in the application.

[0087] In one example, the decision is that an object has been detected. In this case, the location of the scanning window in an image frame (eg, image frame 105) is stored in memory.

[0088] One or more of the embodiments of the present invention may include a vision processing unit (which may be a component of a computer vision module). In one example, a vision processing unit is a class of microprocessors; it is a specific type of AI (artificial intelligence) accelerator designed to accelerate machine vision tasks. The difference between a vision processing unit and a video processing unit (which is specifically used for video encoding and decoding) is that it is suitable for running machine vision algorithms, such as convolutional neural networks, SIFT, etc. The vision processing unit may include a direct interface for acquiring data from a camera (bypassing any off-chip buffers) and may place more emphasis on on-chip data flow between many parallel execution units with registers (such as a multi-core DSP (digital signal processor)). However, like the video processing unit, the vision processing unit may focus on low-precision fixed-point algorithms for image processing.

[0089] In one embodiment, the recognition algorithms can be divided into two main approaches: geometry, which focuses on distinguishing features, or photometry, which is a statistical method that extracts the image into values ​​and compares them to a template to eliminate differences.

[0090] Some of the recognition algorithms include principal component analysis using eigenfaces, linear discriminant analysis, elastic bundle graph matching using Fisherface algorithm, hidden Markov models, multilinear subspace learning using tensor representation, and neuron-excitation dynamic link matching.

[0091] Additionally or alternatively, a computing device (e.g., a device of vehicle 101) may also transmit the captured audio / video to a computing device such as a back-end server via a network for subsequent processing thereof. In some cases, this back-end server may perform audio, image, and / or video analysis to determine the presence or absence of an object or event. If detection is performed, the back-end server may transmit a message to a computing device, a client device, and / or other devices. In some cases, the transmitted audio / video may be stored in a storage device (e.g., a database) to be used as training data for generating and / or improving a classifier or machine learning tool.

[0092] Computing device (e.g. Figure 1 The vehicle 101 may include components that facilitate the operation of the camera. For example, the imager may include a video recording sensor and / or a camera chip. In one aspect of the invention, the imager may include a complementary metal oxide semiconductor (CMOS) array and may be capable of recording high definition (e.g., 720p, 2120p, etc.) video files.

[0093] The camera processor may include an encoding and compression chip. In some embodiments, the camera processor may include a bridge processor. The camera processor may process the video recorded by the imager and may convert this data into a form suitable for wireless transmission to the network through the communication module. The camera memory may include volatile memory that can be used when buffering or encoding data by the camera processor. For example, in some embodiments, the camera memory may include synchronous dynamic random access memory (SDRAM). (Several) infrared LEDs may include light emitting diodes capable of radiating infrared light. (Several) infrared filters may include a system that configures the imager to primarily see infrared light rather than visible light when triggered. When (several) light sensors detect low levels of ambient light (which may include levels that hinder imager performance in the visible spectrum), (several) infrared LEDs may emit infrared light into the environment, and (several) infrared filters may enable the imager to see this infrared light because it is reflected or refracted from an object within the sensor's field of view. This process can provide night vision capabilities for the computing device.

[0094] Various embodiments related to using image frames to scan image data may be implemented using a computing device (e.g., vehicle 101) that includes processor(s) (e.g., processor 107), memory (e.g., memory 109), display(s), communication module(s), input device(s), speaker(s), microphone(s), connector(s), battery, and data port. These components are communicatively coupled together via an interconnect bus. The processor(s) may include any processor used in smartphones and / or portable computing devices, such as an ARM processor (a processor based on the RISC (Reduced Instruction Set Computer) architecture developed by Advanced RISC Machines (ARM)). In some embodiments, the processor(s) may include one or more other processors (e.g., one or more conventional microprocessors) and / or one or more auxiliary coprocessors (e.g., math coprocessors).

[0095] The memory may include operating memory, such as random access memory (RAM), and data storage devices, such as read-only memory (ROM), hard disk drives, flash memory, or any other suitable memory / storage elements. The memory may include removable memory elements, such as compact flash memory cards, multimedia cards (MMC), and / or secure digital (SD) cards. In some embodiments, the memory may include a combination of magnetic, optical, and / or semiconductor memory, and may include, for example, RAM, ROM, flash memory drives, and / or hard disks or drives. The processor(s) and memory may each be, for example, completely located within a single device, or may be connected to each other via communication media such as a USB port, a serial port cable, a coaxial cable, an Ethernet-type cable, a telephone line, a radio frequency transceiver, or other similar wireless or wired media, or a combination of the foregoing. For example, the processor(s) may be connected to the memory via a data port.

[0096] The display(s) may include any user interface or presentation element suitable for a smartphone and / or portable computing device, such as a keypad, display screen, touch screen, microphone, and speaker. The communication module(s) are configured to handle the communication link between the computing device and other external devices (e.g., servers) or receivers, and to route incoming / outgoing data appropriately. For example, inbound data from a data port may be routed through the communication module(s) before being directed to the processor(s), and outbound data from the processor(s) may be routed through the communication module(s) before being directed to the data port. The communication module(s) may include one or more transceiver modules capable of transmitting and receiving data, and using, for example, one or more protocols and / or technologies, such as GSM, UMTS (3GSM), IS-95 (CDMA one), IS-2000 (CDMA 2000), LTE, FDMA, TDMA, W-CDMA, CDMA, OFDMA, Wi-Fi, WiMAX, or any other protocols and / or technologies.

[0097] The data port may be any type of connector for physically interfacing with a smartphone and / or portable computing device, such as a mini-USB or USB-C port or 30-pin connector or In other embodiments, the data port may include multiple communication channels for simultaneously communicating with, for example, other processors, servers, and / or client terminals.

[0098] The memory may store instructions for communicating with other systems such as computers. The memory may store, for example, programs (e.g., computer program codes) suitable for guiding (several) processors according to embodiments of the present invention. The instructions may also include program elements, such as an operating system. Although the execution of the sequence of instructions in the program causes (several) processors to perform the process steps described herein, hard-wired circuits may be used instead of or in combination with software / firmware instructions to implement the processes of embodiments of the present invention. Therefore, embodiments of the present invention are not limited to any particular combination of hardware and software.

[0099] The input device(s) may include any combination of hardware and / or software elements that receive user input and / or facilitate interpretation of received input. The input device(s) may be hardware (e.g., physical buttons and switches), software (e.g., virtual buttons, sliders, etc.), or some combination thereof (e.g., software-controlled tactile feedback to simulate button clicks). In some cases, the input device(s) include a touch screen or a touch-based digitizer. The input device(s) may also include a virtual keyboard or other virtual input element.

[0100] The speaker(s) may include any combination of speakers or other sound-generating devices. The speaker(s) may be operable to produce a variety of sounds, such as audio from live video, notification or alarm sounds, or other audible sounds.

[0101] The microphone(s) may include any combination of transducers that convert pressure waves into electrical signals. The microphone(s) may capture audio, which in some cases may be transmitted to a separate computing device or server. That transmitted audio may then be relayed to another computing device.

[0102] The antenna may enable the computing device to communicate wirelessly. For example, the antenna allows the computing device to communicate over a cellular network via one or more communication standards (e.g., GSM, CDMA, LTE, etc.). The antenna may allow the computing device to communicate over other wireless protocols, such as Wi-Fi or Bluetooth, among other wireless protocols. Depending on the particular implementation, the antenna may include multiple antennas.

[0103] Figure 4 An autonomous vehicle 610 is shown including sensor modules 612, 614 providing data to a central processing device 618 according to one embodiment. The central processing device 618 is Figure 3 An instance of processor 107 .

[0104] Each sensor module 612, 614 provides object data (e.g., data about nearby vehicles) to a central processing device 618. One or more other sensors 616 provide sensor data to the central processing device 618. In one embodiment, the sensors 616 provide raw sensor data and / or object data.

[0105] The central processing device 618 compares the object data received from each of the sensor modules 612, 614. In one example, the comparison is based on detecting a single object in the field of view of the sensor module 612 and the sensor module 614. In one example, the comparison is based on a correlation determined from a number of events, each event being associated with the detection of a different object. The central processing device 618 uses the comparison of the object data to determine whether the object data from the sensor module 612 corresponds to an object.

[0106] In response to detecting the object, central processing device 618 sends a signal to host processing device 620. The signal indicates that the object has been identified.

[0107] In one embodiment, the host processing device 620 performs one or more actions in response to receiving a signal indicating object detection. In various embodiments, the host processing device 620 sends control signals that change the operation of one or more aspects of the autonomous vehicle 610 to one or more vehicle systems 622. In one example, the field of view of a camera acting as a virtual mirror is changed. In one example, navigation, brake control, and / or engine control are changed. In another example, the configuration of one or more vehicle systems 622 is updated. In one example, firmware for one of the vehicle systems 622 is updated. In one example, at least a portion of the firmware is downloaded from a central server using wireless communication.

[0108] In one embodiment, the sensor 616 provides data used to determine the operating environment of the autonomous vehicle 610 to the central processing device 618. For example, the sensor 616 may provide data indicating whether the vehicle is currently operating during the day or at night. This provides environmental data that the central processing device 618 can use to analyze the image data to detect objects. In one embodiment, the environmental data is used to select the field of view of the cameras 150, 152.

[0109] In one example, the central processing device 618 is a system on a chip (SOC), a field programmable gate array (FPGA), a CPU, or a graphics processing unit (GPU). In one example, the central processing device 618 stores runtime data in a volatile memory device(s) (e.g., a DRAM device). In one example, the central processing device 618 may include logic circuitry that implements at least a portion of the various embodiments described herein.

[0110] In one example, the host processing device 620 is a system on a chip (SOC), a field programmable gate array (FPGA), a CPU, or a graphics processing unit (GPU). In one example, the host processing device 620 stores data in a volatile memory device(s). For example, the stored data is received from the vehicle system 622. The host processing device 620 can be, for example, a processing core of a processor, an execution unit, etc.

[0111] In one example, the central processing device 618 and / or the host processing device 620 may include hardware such as one or more integrated circuits and / or discrete components, buffer memory, cache memory, or a combination thereof. The host or central processing device may be a microcontroller, a dedicated logic circuit (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or another suitable processor.

[0112] In one example, each sensor module 612, 614 is a camera. Each camera shares a common or partially overlapping field of view. In one example, each camera is mounted on the same side of the vehicle. In one example, it is expected that during normal operation, the cameras will detect the same object most of the time.

[0113] In an alternative embodiment, the sensor modules 612, 614 are cameras used on a vehicle controlled by a driver. In one example, the cameras work together to provide a virtual mirror for the driver.

[0114] Figure 5 A sensor module 612 is shown including a sensor 624 that provides raw data to a processor 626 according to one embodiment. The processor 626 is Figure 3 An instance of processor 107 .

[0115] In one example, sensor 624 is a camera. In another example, sensor 624 is a radar or lidar sensor. Processor 626 is, for example, a microprocessor, a GPU, and / or an FPGA.

[0116] Processor 626 performs one or more processes to evaluate the raw data from sensor 624 (e.g., to detect nearby vehicles). Processor 626 provides object data as a result of this processing. The object data is transmitted to a central processing device. The object data includes, for example, location data and / or object type data. Other metadata that characterizes the detected object may also be included.

[0117] During operation, the processor 626 accesses the memory 628. The memory 628 is, for example, a volatile memory and / or a non-volatile memory.

[0118] The memory 628 includes an object database 630. The database 630 includes a library of object types. When providing the results of evaluating the raw data from the sensor 624, the processor 626 can retrieve predetermined object types from the library.

[0119] In one embodiment, the object type is displayed Figure 2 on the display screen 172.

[0120] In one embodiment, neural network 632 is stored in memory 628. Neural network 632 may be used to provide one or more inputs to processor 626 while processor 626 is evaluating raw data from sensor 624. In one embodiment, neural network 632 receives inputs from sensor 616 for determining the operating environment of autonomous vehicle 610 as outputs from neural network 632. The outputs are provided to processor 626 for use in evaluating raw data from sensor 624. The results of the evaluation may be displayed to the driver on display screen 172.

[0121] Figure 6 An approaching vehicle 300 is shown overtaking a vehicle 200 driven by a driver, according to one embodiment. The computer system of vehicle 200 detects the presence of vehicle 300 within predetermined zone 654.

[0122] In one embodiment, the vehicle 300 is detected based on determining that the distance 650 of the vehicle 300 behind the vehicle 200 is less than the predetermined distance 652. In response to detecting the vehicle 300, one or more cameras of the vehicle 200 are adjusted to change their field of view. In one example, the camera is mounted on the side of the vehicle 200. In one example, the camera is mounted at the rear of the vehicle 200.

[0123] Figure 7 An approaching vehicle 200 is shown in a different lane from the driver's vehicle 300 and preparing to pass, according to one embodiment. Vehicle 300 detects the presence of vehicle 200, such as described above. In response to detecting vehicle 200, the field of view of camera 742 is adjusted from narrow angle 740 to wide angle 741.

[0124] Figure 8Vehicle 300 is shown approaching from behind the driver's vehicle 200 and then changing lanes in preparation for passing, according to one embodiment. Vehicle 200 detects vehicle 300 based on determining that distance 804 of vehicle 300 behind vehicle 200 is less than a predetermined distance. Vehicle 200 also determines that vehicle 300 is traveling at a speed greater than vehicle 200. In response to detecting the presence of vehicle 300 and determining the speed of vehicle 300, vehicle 200 adjusts the field of view of camera 806 to have angle 802.

[0125] exist Fig. 9 In FIG. 8 , a computing device 8201 includes an interconnect 8202 (eg, a bus and system core logic) that interconnects (several) microprocessors 8203 and memory 8208. In FIG. Fig. 9 In the example, microprocessor 8203 is coupled to cache memory 8204.

[0126] The interconnect 8202 interconnects the microprocessor(s) 8203 and memory 8208 together and also interconnects them through the input / output controller(s) 8206 to a display controller and display device 8207 and to peripheral devices such as input / output (I / O) devices 8205. Typical I / O devices include a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices well known in the art.

[0127] The interconnect 8202 may include one or more buses connected to each other through various bridges, controllers and / or adapters. In one embodiment, the I / O controller 8206 includes a USB (Universal Serial Bus) adapter for controlling USB peripheral devices, and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripheral devices.

[0128] The memory 8208 may include ROM (Read Only Memory) as well as volatile RAM (Random Access Memory) and non-volatile memory such as a hard disk drive, flash memory, and the like.

[0129] Volatile RAM is typically implemented as dynamic RAM (DRAM), which continuously requires power in order to refresh or maintain the data in the memory. Nonvolatile memory is typically a solid-state drive, a magnetic hard drive, a magnetic optical drive, or an optical drive (e.g., DVD RAM), or other types of memory systems that can maintain data even after power is removed from it. Nonvolatile memory can also be random access memory.

[0130] The non-volatile memory may be a local device directly coupled to the rest of the components in the computing device. Non-volatile memory remote from the computing device may also be used, such as a network storage device coupled to the computing device through a network interface (e.g., a modem or Ethernet interface).

[0131] In one embodiment, Fig. 9 The computing device described in is used to implement a computing device including a computer 131, a processor 107, a sensor module 612, a central processing device 618 and / or a host processing device 620.

[0132] In another embodiment, if Fig. 9 The computing device described in the embodiment is used to implement a user terminal or mobile device on which an application is installed or being installed. The user terminal may be in the form of a laptop or notebook computer or a personal desktop computer, for example.

[0133] In some embodiments, one or more servers may be replaced with the services of a peer-to-peer network of multiple data processing systems or a network of distributed computing systems. The peer-to-peer network or distributed computing systems may be collectively considered as computing devices.

[0134] Embodiments of the present invention may be implemented via microprocessor(s) 8203 and / or memory 8208. For example, the described functions may be implemented in part via hardware logic in microprocessor(s) 8203 and in part using instructions stored in memory 8208. Some embodiments are implemented using microprocessor(s) 8203 without additional instructions stored in memory 8208. Some embodiments are implemented using instructions stored in memory 8208 for execution by one or more general purpose microprocessors 8203. Thus, the present invention is not limited to a particular configuration of hardware and / or software.

[0135] Fig.10 A block diagram of a computing device (eg, a user's mobile device or a user terminal) according to one embodiment is shown. In one example, the computing device displays an image based on image data collected by camera 150 or 152 .

[0136] exist Fig.10 In the embodiment, the computing device includes an interconnect 9221 connecting a presentation device 9229, a user input device 9231, a processor 9233, a memory 9227, a location identification unit 9225 and a communication device 9223.

[0137] exist Fig.10 In the embodiment of the present invention, the location identification unit 9225 is used to identify the geographic location. The location identification unit 9225 may include a satellite positioning system receiver, such as a global positioning system (GPS) receiver, to automatically identify the current location of the computing device.

[0138] exist Fig.10 In one embodiment, the communication device 9223 is configured to communicate with the server to provide data, including parameter data. In one embodiment, the user input device 9231 is configured to receive or generate user data or content. The user input device 9231 may include a text input device, a still image camera, a video camera and / or a recorder, etc.

[0139] Fig.11 A method for adjusting the field of view of a vehicle camera based on detection of nearby vehicles is presented according to one embodiment. For example, Fig.11 The method can be found in Figure 1 and 2 implemented in the system.

[0140] Fig.11 The method may be performed by processing logic that may include hardware (e.g., a processing device, a circuit, dedicated logic, programmable logic, microcode, device hardware, integrated circuits, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof.

[0141] In some embodiments, Fig.11 The method may be performed at least in part by one or more processors. In some embodiments, the method is performed using Fig. 9 or 10 processors and memories.

[0142] Although shown in a particular order or sequence, the order of the processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0143] At block 1101, image data acquired using a first field of view of a camera mounted on the first vehicle is displayed to a user of a first vehicle. In one example, the image data is acquired using camera 150 having a field of view having an angle 154. In one example, the image data is acquired using Figure 7 The camera 742 obtains the image data. The camera 742 obtains the image data using a field of view having a narrow angle 740.

[0144] At block 1103, a second vehicle is detected by a first vehicle. In one example, the first vehicle is vehicle 7111 and the second vehicle is object 7157. In one example, the first vehicle is Figure 7 The second means of transport is Figure 7 200 means of transportation.

[0145] At block 1105, it is determined whether the second vehicle is within a predetermined zone relative to the first vehicle. In one example, vehicle 7111 determines whether object 7157 is within predetermined zone 164. In one example, vehicle 7111 determines whether object 7155 is within zone 162.

[0146] At block 1107, in response to determining that the second vehicle is within the predetermined zone, image data obtained using the second field of view of the camera is displayed to the user. Figure 7 The vehicle 300 obtains image data from a camera 742 using a field of view having a wide angle 741 .

[0147] In one embodiment, a method includes: providing a first vehicle (e.g., Figure 7 The method of claim 1 , wherein the at least one processing device of the first vehicle detects a second vehicle (e.g., a plurality of vehicles) that are located outside the first vehicle, and the at least one processing device of the first vehicle detects a second vehicle (e.g., a plurality of vehicles) that are located outside the first vehicle, and the plurality of vehicles ... Figure 7 vehicle 200); determining, by the one processing device, whether the second vehicle is within a predetermined area relative to the first vehicle; and in response to determining that the second vehicle is within the predetermined area, displaying image data obtained using a second field of view (e.g., wide angle 741) of the camera.

[0148] In one embodiment, the first field of view is narrower than the second field of view.

[0149] In one embodiment, the second field of view includes a first region on the side of the first vehicle, the first region including a region associated with a user's blind spot when operating the vehicle.

[0150] In one embodiment, determining whether the second vehicle is within the predetermined zone includes determining whether the second vehicle is within a predetermined distance (e.g., Figure 6 within the distance 652).

[0151] In one embodiment, displaying image data acquired using the second field of view is further responsive to determining that a speed of the second vehicle is greater than a speed of the first vehicle.

[0152] In one embodiment, the method further comprises: determining whether the first vehicle is traveling faster than a predetermined speed; and wherein displaying the image data obtained using the first field of view is responsive to determining that the first vehicle is traveling faster than the predetermined speed.

[0153] In one embodiment, the camera is mounted on the side of the first vehicle, and the method further includes: determining whether there is a vehicle on the side of the first vehicle; wherein displaying the image data obtained using the first field of view is in response to determining that there is no vehicle on the side of the first vehicle.

[0154] In one embodiment, the camera is mounted on a side of the first vehicle, and the predetermined area is an area on the side of the first vehicle, or an area behind the first vehicle and within a predetermined distance of the first vehicle.

[0155] In one embodiment, the method further includes: determining that the second vehicle has passed the first vehicle; in response to determining that the second vehicle has passed the first vehicle, changing the configuration of the camera to collect image data using the first field of view.

[0156] In one embodiment, changing the configuration of the camera to collect image data using the first field of view is further responsive to determining that no vehicles are present in a lateral region of the first vehicle in which a second vehicle has passed the first vehicle.

[0157] In one embodiment, detecting the second vehicle includes receiving, via the first vehicle, a communication from the second vehicle.

[0158] In one embodiment, displaying image data acquired using the second field of view of the camera is further responsive to determining that the second vehicle is moving in the same direction as the first vehicle.

[0159] In one embodiment, determining whether the second vehicle is within the predetermined zone comprises determining whether the second vehicle is within a predetermined viewing angle relative to the position of the camera.

[0160] In one embodiment, wherein the location is a first location (e.g., the location of a camera), and wherein determining whether the second vehicle is within a predetermined area further comprises determining whether the second vehicle is within a predetermined distance of the first location or within a predetermined distance of a second location (e.g., a rear location) of the first vehicle.

[0161] In one embodiment, a system includes: a camera (e.g., camera 170) mounted on a side of a first vehicle and configured to collect image data about objects external to the first vehicle; a display screen (e.g., display screen 172) positioned for viewing by a driver of the first vehicle; at least one processing device; and a memory containing instructions configured to instruct at least one processing device to: present an image on the display screen based on data collected using a first field of view of the camera; detect a second vehicle; determine whether the second vehicle is within a predetermined area; and in response to determining that the second vehicle is within the predetermined area, present an image on the display screen based on data collected using a second field of view of the camera.

[0162] In one embodiment, the first field of view is narrower than the second field of view.

[0163] In one embodiment, when the second vehicle is within a predetermined distance of the first vehicle, it is determined that the second vehicle is within the predetermined zone.

[0164] In one embodiment, detecting the second vehicle includes receiving a communication from the second vehicle, the communication including the location of the second vehicle; and determining whether the second vehicle is within the predetermined area includes determining whether the location of the second vehicle is within a predetermined distance of the location of the first vehicle.

[0165] In one embodiment, presenting the image based on data collected using the first field of view is responsive to determining that the first vehicle is traveling faster than a predetermined speed and is further responsive to determining that there are no vehicles to the side of the first vehicle.

[0166] In one embodiment, a non-transitory computer storage medium stores instructions that, when executed on at least one processing device, cause the at least one processing device to at least: display image data obtained using a first field of view of a camera of the first vehicle to a user of a first vehicle; detect a second vehicle; determine whether the second vehicle is within a predetermined area relative to the first vehicle; and in response to determining that the second vehicle is within the predetermined area, display image data obtained using a second field of view of the camera.

[0167] Fig.12 is a block diagram of a vehicle according to some embodiments. Fig.12 The system described in can be installed completely within a vehicle.

[0168] The system optionally includes an autonomous vehicle subsystem 702. In the illustrated embodiment, the autonomous vehicle subsystem 702 includes a map database 702A, a radar device 702B, a laser radar device 702C, a digital camera 702D, a sonar device 702E, a GPS receiver 702F, and an inertial measurement unit 702G. In one embodiment, the map database 702A stores multiple high-definition three-dimensional maps for path selection and navigation. The radar device 702B, the laser radar device 702C, the digital camera 702D, the sonar device 702E, the GPS receiver 702F, and the inertial measurement unit 702G may include various corresponding devices installed at various locations throughout the autonomous vehicle. For example, these devices can be installed along the periphery of the autonomous vehicle to provide position perception, collision avoidance, and other standard autonomous vehicle functions. In some embodiments, the autonomous subsystem 702 is built into the vehicle, and in other embodiments, the autonomous subsystem 702 includes an aftermarket system.

[0169] Vehicle subsystem 706 is also included in the system. Vehicle subsystem 706 includes various anti-lock braking systems 706A, engine control unit 706B, and transmission control unit 706C. These components can be used to control the operation of the vehicle in response to data generated by autonomous vehicle subsystem 702 and / or ADAS subsystem 704.

[0170] The processing side of the system includes one or more processors 710 , short-term memory 712 , an RF system 714 , a graphics processing unit (GPU) 716 , long-term storage 718 , and one or more interfaces 720 .

[0171] The one or more processors 710 may include a central processing unit, an FPGA, or any range of processing devices required to support the operation of the autonomous vehicle. The memory 712 includes a DRAM or other suitable volatile RAM for temporarily storing data required by the processor 710. The RF system 714 may include a cellular transceiver and / or a satellite transceiver. The long-term storage device 718 may include one or more high-capacity solid-state drives (SSDs). In general, the long-term storage device 718 can be used to store, for example, high-definition maps, path selection data, and any other data that needs permanent or semi-permanent storage. The GPU 716 may include one or more high-throughput GPU devices for processing data received from the autonomous vehicle subsystem 702A. Finally, the interface 720 may include various display units (e.g., dashboard screens) positioned within the vehicle.

[0172] In one embodiment, the system further includes an ADAS subsystem 704, which can perform at least a portion of the operations described by the method illustrated in the previous figure. The ADAS subsystem 704 includes a blind spot ADAS 704a and a distance ADAS 704b that execute a blind spot algorithm to detect nearby vehicles and adjust the camera field of view. The distance detection algorithm determines the distance of nearby vehicles behind the driver's vehicle. The outputs of these systems control the components of the vehicle subsystem 706. In addition, the ADAS subsystem 704 includes an authentication system 704c, which may include higher-level software / circuitry.

[0173] Each of the devices is connected via a bus 708. In one embodiment, the bus 708 may include a controller area network (CAN) bus. In some embodiments, other bus types (e.g., FlexRay or MOST buses) may be used. In addition, each subsystem may include one or more additional buses to handle inter-subsystem communications (e.g., a LIN bus for lower bandwidth communications).

[0174] Conclusion

[0175] The present invention includes various devices that perform the methods described above and implement the systems described above, including data processing systems that perform these methods, and computer-readable media containing instructions that, when executed on a data processing system, cause the system to perform these methods.

[0176] The description and drawings are illustrative and should not be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described in order to avoid obscuring the description. References to one embodiment or an embodiment in the present invention are not necessarily references to the same embodiment; and such references represent at least one embodiment.

[0177] References in this specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not others. Similarly, various requirements are described that may be requirements of some embodiments but not others.

[0178] As used herein, “coupled to” generally refers to a connection between components, which may be an indirect communication connection, wired or wireless, or a direct communication connection (eg, without intermediate components), including, for example, an electrical connection, an optical connection, a magnetic connection, etc.

[0179] In this description, various functions and operations may be described as being performed or caused by software code to simplify the description. However, those skilled in the art will recognize that such expressions mean that the functions result from the execution of code by one or more processors (e.g., microprocessors, application specific integrated circuits (ASICs), graphics processors, and / or field programmable gate arrays (FPGAs)). Alternatively or in combination, dedicated circuits (e.g., logic circuits) with or without software instructions may be used to implement the functions and operations. Embodiments may be implemented using hardwired circuits without software instructions or using hardwired circuits in combination with software instructions. Therefore, the technology is not limited to any specific combination of hardware circuits and software, nor is it limited to any specific source of instructions executed by a computing device.

[0180] Although some embodiments may be implemented in fully functional computers and computer systems, the various embodiments are capable of being distributed as a computer product in a variety of forms and can be used regardless of the specific type of machine or computer readable media used to actually achieve the distribution.

[0181] At least some of the disclosed aspects may be embodied at least in part in software. That is, the techniques may be implemented in a computing device or other system in response to its processor (e.g., a microprocessor) executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, nonvolatile memory, cache memory, or remote storage device).

[0182] The routines executed to implement the embodiments may be implemented as part of an operating system, middleware, service delivery platform, SDK (software development kit) component, network service or other specific application, component, program, object, module or sequence of instructions called a "computer program". The calling interface of these routines may be exposed to the software development community as an API (application programming interface). A computer program typically includes one or more instructions that are set at different times in various memories and storage devices in a computer and that, when read and executed by one or more processors in a computer, cause the computer to perform operations necessary to implement the elements involved in various aspects.

[0183] Machine-readable media can be used to store software and data that, when executed by a computing device, causes the device to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache memory. Portions of this software and / or data can be stored in any of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or in the same communication session. Data and instructions can be obtained as a whole before executing an application. Alternatively, portions of data and instructions can be obtained dynamically and in a timely manner when execution requires. Therefore, at a specific time instance, data and instructions do not need to be entirely located on machine-readable media.

[0184] Examples of computer-readable media include, but are not limited to, recordable and non-recordable types of media, such as volatile and non-volatile storage devices, read-only memory (ROM), random access memory (RAM), flash memory devices, solid-state drive storage media, removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CDROM), digital versatile disk (DVD), etc.), etc. A computer-readable medium may store instructions.

[0185] Generally speaking, a tangible or non-transitory machine-readable medium includes any mechanism that provides (e.g., stores) information in a form accessible by a machine (e.g., a computer, mobile device, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.).

[0186] In various embodiments, software instructions may be combined using hardwired circuits to implement the techniques. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source of instructions executed by a computing device.

[0187] Although some of the figures illustrate several operations in a particular order, order-independent operations may be reordered and other operations may be combined or decomposed. Although some reordering or other groupings are specifically mentioned, other groupings will be apparent to one of ordinary skill in the art and therefore no exhaustive list of alternatives is presented. Furthermore, it should be recognized that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0188] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made to the invention without departing from the broader spirit and scope as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0189] The various embodiments described herein can be implemented using various different types of computing devices. As used herein, examples of "computing devices" include, but are not limited to, servers, centralized computing platforms, systems with multiple computing processors and / or components, mobile devices, user terminals, vehicles, personal communication devices, wearable digital devices, electronic information terminals, general-purpose computers, electronic document readers, tablet computers, laptop computers, smart phones, digital cameras, household appliances, televisions, or digital music players. Additional examples of computing devices include devices that are part of the "Internet of Things" (IOT). Such "things" may occasionally interact with their owners or administrators, who may monitor things or modify the settings of these things. In some cases, such owners or administrators play the role of users with respect to "thing" devices. In some instances, with respect to paired "thing" devices (e.g., Apple watches) worn by users, the user's primary mobile device (e.g., Apple iPhone) may be an administrator server.

[0190] In some embodiments, the computing device may be a host system implemented as, for example, a desktop computer, a laptop computer, a network server, a mobile device, or other computing device including a memory and a processing device. The host system may include or be coupled to a memory subsystem such that the host system can read data from or write data to the memory subsystem. The host system may be coupled to the memory subsystem via a physical host interface.

[0191] Examples of physical host interfaces include, but are not limited to, a Serial Advanced Technology Attachment (SATA) interface, a Peripheral Component Interconnect Express (PCIe) interface, a Universal Serial Bus (USB) interface, Fibre Channel, Serial Attached SCSI (SAS), a Double Data Rate (DDR) memory bus, and the like. The physical host interface may be used to transfer data between a host system and a memory subsystem. When a memory subsystem is coupled to a host system via a PCIe interface, the host system may further utilize an NVM Express (NVMe) interface to access memory components of the memory subsystem. The physical host interface may provide an interface for passing control, address, data, and other signals between a memory subsystem and a host system. In general, a host system may access multiple memory subsystems via the same communication connection, multiple separate communication connections, and / or a combination of communication connections.

[0192] In one embodiment, the host system includes a processing device and a controller. The processing device of the host system may be, for example, a microprocessor, a graphics processing unit, a central processing unit (CPU), an FPGA, a processing core of a processor, an execution unit, etc. In one example, the processing device may be a single package that combines an FPGA and a microprocessor, wherein the microprocessor performs most of the processing, but pushes certain predetermined specific tasks to the FPGA module. In one example, the processing device is a soft microprocessor (sometimes also referred to as a soft-core microprocessor or soft processor), which is a microprocessor core implemented using logic synthesis. The soft microprocessor can be implemented via different semiconductor devices containing programmable logic (e.g., ASIC, FPGA, or CPLD).

[0193] In some examples, the controller is a memory controller, a memory management unit, and / or an initiator. In one example, the controller controls communications over a bus coupled between a host system and a memory subsystem.

[0194] In general, the controller may send commands or requests to the memory subsystem to obtain desired access to the memory component. The controller may further include interface circuitry that communicates with the memory subsystem. The interface circuitry may convert responses received from the memory subsystem into information for the host system. The controller of the host system may communicate with the controller of the memory subsystem to perform operations such as reading data, writing data, or erasing data at the memory component and other such operations.

[0195] In some cases, the controller may be integrated with the processing device in the same package. In other cases, the controller is separated from the packaging of the processing device. The controller and / or the processing device may include hardware such as one or more integrated circuits and / or discrete components, buffer memory, cache memory, or a combination thereof. The controller and / or the processing device may be a microcontroller, a dedicated logic circuit (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), or another suitable processor.

[0196] The memory components may include any combination of different types of non-volatile memory components and / or volatile memory components. Examples of non-volatile memory components include "NAND" (NAND) type flash memory. Each of the memory components may include one or more memory cell arrays, such as single-level cells (SLC) or multi-level cells (MLC) (e.g., three-level cells (TLC) or four-level cells (QLC)). In some embodiments, a particular memory component may include both an SLC portion and an MLC portion of a memory cell. Each of the memory cells may store one or more data bits (e.g., data blocks) used by a host system. Although non-volatile memory components such as NAND-type flash memory are described, the memory components may be based on any other type of memory such as volatile memory.

[0197] In some embodiments, the memory component may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a phase change memory (PCM), a magnetic random access memory (MRAM), a spin transfer torque (STT)-MRAM, a ferroelectric random access memory (FeTRAM), a ferroelectric RAM (FeRAM), a conductive bridge RAM (CBRAM), a resistive random access memory (RRAM), an oxide-based RRAM (OxRAM), a "NOR" (NOR) flash memory, an electrically erasable programmable read-only memory (EEPROM), a nanowire-based nonvolatile memory, a memory incorporating memristor technology, and a cross-point array of nonvolatile memory cells. The cross-point array of nonvolatile memory can perform bit storage based on changes in bulk resistance in combination with a stackable cross-grid data access array. In addition, compared to many flash-based memories, the cross-point nonvolatile memory can perform write-in-place operations, where nonvolatile memory cells can be programmed without having to erase the nonvolatile memory cells in advance. Furthermore, memory cells of a memory component may be grouped into memory pages or data blocks, which may refer to units of the memory component for storing data.

[0198] The controller of the memory subsystem may communicate with the memory components to perform operations such as reading data, writing data, or erasing data at the memory components and other such operations (e.g., in response to commands dispatched by the controller on a command bus). The controller may include a processing device (processor) configured to execute instructions stored in a local memory. The local memory of the controller may include an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines for controlling the operation of the memory subsystem, including handling communications between the memory subsystem and the host system. In some embodiments, the local memory may include memory registers that store memory pointers, extracted data, etc. The local memory may also include a read-only memory (ROM) for storing microcode. Although the example memory subsystem includes a controller, in another embodiment of the present invention, the memory subsystem may not include a controller, but may rely on external control (e.g., provided by an external host or by a processor or controller separate from the memory subsystem).

[0199] In general, the controller may receive commands or operations from the host system and may convert the commands or operations into instructions or appropriate commands to achieve the desired access to the memory component. The controller may be responsible for other operations associated with the memory component, such as wear leveling operations, garbage collection operations, error detection and error correction code (ECC) operations, encryption operations, cache operations, and address translation between logical block addresses and physical block addresses. The controller may further include host interface circuitry to communicate with the host system via a physical host interface. The host interface circuitry may convert commands received from the host system into command instructions to access the memory component and convert responses associated with the memory component into information for the host system.

[0200] The memory subsystem may also include additional circuits or components not illustrated. In some embodiments, the memory subsystem may include a cache or buffer (e.g., DRAM or SRAM) and address circuits (e.g., row decoders and column decoders) that can receive addresses from a controller and decode addresses to access memory components.

Claims

1. A method for adjusting the field of view of a vehicle camera, the method comprising: In response to determining that a first vehicle is traveling faster than a predetermined speed, displaying image data obtained using a first field of view of a camera mounted on a side of the first vehicle to a user of the first vehicle, wherein the camera collects the image data of objects located outside of the first vehicle; detecting a second vehicle by at least one processing device of the first vehicle; determining, by the one processing device, whether the second vehicle is within a predetermined zone relative to the first vehicle; and In response to determining that the second vehicle is within the predetermined zone, image data acquired using a second field of view of the camera is displayed, wherein the first field of view is narrower than the second field of view. 2 . The method of claim 1 , wherein the second field of view includes a first region on a side of the first vehicle, the first region including a region associated with a blind spot of the user when operating the vehicle. 3 . The method of claim 1 , wherein determining whether the second vehicle is within the predetermined zone comprises determining whether the second vehicle is within a predetermined distance of the first vehicle. 4 . The method of claim 1 , wherein displaying the image data acquired using the second field of view is further responsive to determining that a speed of the second vehicle is greater than a speed of the first vehicle.

5. The method according to claim 1, further comprising: determining whether there is a vehicle on the side of the first vehicle; Wherein displaying the image data obtained using the first field of view is further responsive to determining that no vehicle is present to the side of the first vehicle. 6 . The method according to claim 1 , wherein the predetermined area is an area on the side of the first vehicle, or an area behind the first vehicle and within a predetermined distance of the first vehicle.

7. The method according to claim 1, further comprising: determining that the second vehicle has passed the first vehicle; In response to determining that the second vehicle has passed the first vehicle, the configuration of the camera is changed to collect image data using the first field of view.

8. The method of claim 7, wherein changing the configuration of the camera to collect image data using the first field of view is further responsive to determining that no vehicles are present in a lateral area of ​​the first vehicle in which the second vehicle has passed the first vehicle.

9. The method of claim 1, wherein detecting the second vehicle comprises receiving, via the first vehicle, a communication from the second vehicle.

10. The method of claim 1, wherein displaying the image data acquired using the second field of view of the camera is further responsive to determining that the second vehicle is moving in the same direction as the first vehicle. 11 . The method of claim 1 , wherein determining whether the second vehicle is within the predetermined zone comprises determining whether the second vehicle is within a predetermined viewing angle relative to the position of the camera.

12. The method of claim 11, wherein the location is a first location, and wherein determining whether the second vehicle is within the predetermined area further comprises determining whether the second vehicle is within a predetermined distance of the first location or within a predetermined distance of a second location of the first vehicle.

13. A transportation system, comprising: a camera mounted on a side of a first vehicle and configured to collect image data regarding objects external to the first vehicle; a display screen positioned for viewing by a driver of the first vehicle; at least one processing device; and A memory containing instructions configured to instruct the at least one processing device to: In response to determining that the first vehicle is traveling faster than a predetermined speed, presenting an image on the display screen based on data collected using a first field of view of the camera; detecting a second means of transport; determining whether the second vehicle is within a predetermined area; and In response to determining that the second vehicle is within the predetermined zone, presenting an image on the display screen based on data collected using a second field of view of the camera, wherein the first field of view is narrower than the second field of view. 14 . The vehicle system of claim 13 , wherein when the second vehicle is within a predetermined distance of the first vehicle, it is determined that the second vehicle is within the predetermined area.

15. The vehicle system of claim 13, wherein: Detecting the second vehicle includes receiving a communication from the second vehicle, the communication including a location of the second vehicle; and Determining whether the second vehicle is within the predetermined zone includes determining whether the location of the second vehicle is within a predetermined distance of the location of the first vehicle. 16 . The vehicle system of claim 13 , wherein presenting the image based on the data collected using the first field of view is further responsive to determining that no vehicle is present to the side of the first vehicle.

17. A non-transitory computer storage medium storing instructions that, when executed on at least one processing device, cause the at least one processing device to perform at least the following operations: In response to determining that a first vehicle is traveling faster than a predetermined speed, displaying image data obtained using a first field of view of a camera mounted on a side of the first vehicle to a user of the first vehicle; detecting a second means of transport; determining whether the second vehicle is within a predetermined zone relative to the first vehicle; and In response to determining that the second vehicle is within the predetermined zone, image data acquired using a second field of view of the camera is displayed, wherein the first field of view is narrower than the second field of view.

Images