Multi-position rolling shutter camera device

Abstract

Systems and methods related to a multi-position roller shutter door-opening camera are provided. A system may include a camera device, an actuator, a processor, and a storage medium. The camera device may be configured to capture an image within a field of view using a rolling shutter operating in a shutter direction. The actuator may be coupled to the imaging device and configured to rotate the imaging device between a first orientation in which the shutter direction includes a first direction and a second orientation in which the shutter direction includes a second direction different from the first direction. When executed by the processor, the instructions may cause the processor to cause the actuator to rotate the camera device between the first orientation and the second orientation.

Description

[0001] Priority application

[0002] This application claims priority to U.S. Patent Application No. 17 / 806,676, filed on June 13, 2022, entitled “MULTIPLE POSITION ROLLING SHUTTER IMAGING DEVICE,” which is incorporated herein by reference in its entirety. Background Art

[0003] The present application relates to an imaging device such as a camera that utilizes a rolling shutter to capture images. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is an example environment in which a vehicle including one or more components of an autonomous system may be implemented;

[0005] Figure 2 is a diagram of one or more systems of a vehicle including an autonomous system;

[0006] Figure 3 yes Figure 1 and Figure 2 a diagram of one or more devices and / or components of one or more systems;

[0007] Figure 4 is a diagram of some components of an autonomous system;

[0008] Figure 5A is a top view of a representative example of a camera device having a horizontal field of view;

[0009] Figure 5B is a side view of a representative example of a camera device having a vertical field of view;

[0010] Fig. 6A is a diagram illustrating representative distortion of an image of a horizontally moving object captured using a camera having a vertical rolling shutter;

[0011] Figure 6B is a diagram illustrating representative distortion of an image of a horizontally moving object captured using a camera having a horizontal rolling shutter;

[0012] Fig. 7A is a top view of an embodiment of a vehicle including three camera devices and corresponding horizontal fields of view in a first orientation;

[0013] Figure 7B is to use Fig. 7A A side view of a vehicle and a camera device in a first orientation illustrated by a corresponding vertical field of view;

[0014] Figure 7C is shown using a camera device rotated to a second orientation and a corresponding horizontal field of view Fig. 7A A top view of the vehicle and the camera device;

[0015] Fig.7D is with a corresponding vertical field of view Figure 7C A side view of a vehicle and a camera device;

[0016] Fig. 8A is an example embodiment of a multi-position rolling shutter camera system in a first orientation;

[0017] Figure 8B An example of a multi-position rolling shutter camera system in a second orientation rotated approximately 90 degrees from a first orientation;

[0018] Figure 8C It is used to control Fig. 8A and Figure 8B Example method of a multi-position rolling shutter camera system;

[0019] Fig. 9A is a diagram illustrating a vehicle having a multi-position rolling shutter camera system oriented in a first orientation and a corresponding vertical field of view; and

[0020] Fig. 9B is an example of a camera system positioned in a second orientation and a corresponding vertical field of view Fig. 9A Diagram of the vehicle and camera system. DETAILED DESCRIPTION

[0021] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent that the embodiments described in the present disclosure can be implemented without these specific details. In some instances, well-known configurations and devices are illustrated in block diagram form to avoid unnecessarily obscuring aspects of the present disclosure.

[0022] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements (such as those representing systems, devices, modules, instruction blocks and / or data elements, etc.) is illustrated. However, those skilled in the art will understand that, unless explicitly described, the specific order or arrangement of schematic elements in the accompanying drawings is not intended to mean that a specific processing order or sequence, or separation of processing is required. In addition, unless explicitly described, the inclusion of schematic elements in the accompanying drawings is not intended to mean that such elements are required in all embodiments, nor is it intended to mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0023] In addition, in the accompanying drawings, connecting elements (such as solid or dotted lines or arrows, etc.) are used to illustrate the connection, relationship or association between or among two or more other schematic elements, and the absence of any such connecting elements is not intended to mean that there can be no connection, relationship or association. In other words, some connections, relationships or associations between elements are not illustrated in the accompanying drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element can be used to represent multiple connections, relationships or associations between elements. For example, if the connecting element represents the communication of a signal, data or instruction (e.g., "software instruction"), it will be understood by those skilled in the art that such an element can represent one or more than one signal path (e.g., bus) that may be needed to affect the communication.

[0024] Although the terms "first", "second" and / or "third", etc. are used to describe various elements, these elements should not be limited by these terms. The terms "first", "second" and / or "third" are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact without departing from the scope of the described embodiments. Both the first contact and the second contact are contacts, but they are not the same contacts.

[0025] The terms used in the specification of the various embodiments described herein are included only for the purpose of describing specific embodiments and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms "a", "an" and "the" are also intended to include plural forms and can be used interchangeably with "one or more than one" or "at least one", unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more than one of the associated listed items. It will also be understood that when the terms "include", "comprise", "have" and / or "have" are used in this specification, the stated features, integers, steps, operations, elements and / or components are specifically stated, but the presence or addition of one or more than one other features, integers, steps, operations, elements, components and / or groups thereof are not excluded.

[0026] As used herein, the terms "communication" and "communicating" refer to at least one of receiving, receiving, transmitting, transmitting and / or providing information (or information represented by, for example, data, signals, messages, instructions and / or commands, etc.). For a unit (e.g., a device, a system, a component of a device or system, and / or a combination thereof) to communicate with another unit, this means that the unit is able to directly or indirectly receive information from the other unit and / or send (e.g., transmit) information to the other unit. This can refer to a direct or indirect connection that is wired and / or wireless in nature. In addition, even if the transmitted information can be modified, processed, relayed and / or routed between the first unit and the second unit, the two units can communicate with each other. For example, even if the first unit passively receives information and does not actively transmit information to the second unit, the first unit can communicate with the second unit. As another example, if at least one intermediary unit (e.g., a third unit located between the first unit and the second unit) processes the information received from the first unit and transmits the processed information to the second unit, the first unit can communicate with the second unit. In some embodiments, a message may refer to a network packet (eg, a data packet, etc.) that includes data.

[0027] As used herein, the term "if" is optionally interpreted to mean "when," "at," "in response to being determined to be," and / or "in response to being detected," etc., depending on the context. Similarly, the phrases "if it is determined" or "if [the stated condition or event] is detected" are optionally interpreted to mean "when determining," "in response to being determined to be" or "when [the stated condition or event] is detected," and / or "in response to being detected," etc., depending on the context. In addition, as used herein, the terms "have," "have," or "possess," etc. are intended to be open-ended terms. Furthermore, unless expressly stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

[0028] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not yet been described in detail in order not to unnecessarily obscure aspects of the embodiments.

[0029] General Overview

[0030] In some aspects and / or embodiments, the systems, methods, and computer program products described herein provide systems and methods related to a multi-position rolling shutter camera or camera. In some instances, the systems and methods described herein can cause the camera to rotate between a first orientation and a second orientation. In the first orientation, the rolling shutter direction of the camera can include a first direction. In the second orientation, the rolling shutter direction can include a second direction different from the first direction. The camera can rotate or otherwise move between the first orientation and the second orientation to beneficially or advantageously maximize the field of view of the camera and / or reduce distortion in an image captured by the camera. Control of the orientation of the camera can be based on, among other things, analysis of the image captured by the camera.

[0031] Advantages of implementations of the systems and methods described herein include, but are not limited to, the ability to move a camera between a first (e.g., horizontal) rolling shutter position or orientation and a second (e.g., vertical) rolling shutter position or orientation. This can allow the same camera to capture different fields of view. A horizontal rolling shutter orientation and a vertical rolling shutter can provide different benefits, and the ability to switch or select between a horizontal rolling shutter orientation and a vertical rolling shutter can enable the same camera to provide both sets of benefits. For example, in some instances, a vertical rolling shutter position can provide a larger horizontal field of view and be better for stereo vision, while a horizontal rolling shutter position can provide reduced distortion when other objects are moving and the shutter orientation is consistent with the rotation of the LiDAR.

[0032] Reference now Figure 1 , illustrates an example environment 100 in which vehicles including autonomous systems and vehicles not including autonomous systems operate. As illustrated, the environment 100 includes vehicles 102a-102n, objects 104a-104n, routes 106a-106n, areas 108, vehicle-to-infrastructure (V2I) devices 110, a network 112, a remote autonomous vehicle (AV) system 114, a fleet management system 116, and a V2I system 118. The vehicles 102a-102n, the vehicle-to-infrastructure (V2I) devices 110, the network 112, the autonomous vehicle (AV) system 114, the fleet management system 116, and the V2I system 118 are interconnected (e.g., establish connections for communication, etc.) via wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a-104n are interconnected with at least one of vehicles 102a-102n, vehicle-to-infrastructure (V2I) devices 110, networks 112, autonomous vehicle (AV) systems 114, fleet management systems 116, and V2I systems 118 via wired connections, wireless connections, or a combination of wired or wireless connections.

[0033] Vehicles 102a-102n (individually referred to as vehicles 102 and collectively referred to as vehicles 102) include at least one device configured to transport goods and / or people. In some embodiments, vehicles 102 are configured to communicate with V2I devices 110, remote AV systems 114, fleet management systems 116, and / or V2I systems 118 via network 112. In some embodiments, vehicles 102 include cars, buses, trucks, and / or trains, etc. In some embodiments, vehicles 102 are similar to vehicles 200 described herein (see Figure 2 ). In some embodiments, vehicles 200 in the set of vehicles 200 are associated with an autonomous queue manager. In some embodiments, vehicles 102 travel along respective routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) as described herein. In some embodiments, one or more vehicles 102 include an autonomous system (e.g., an autonomous system that is the same as or similar to autonomous system 202).

[0034] Objects 104a-104n (individually referred to as objects 104 and collectively referred to as objects 104) include, for example, at least one vehicle, at least one pedestrian, at least one cyclist, and / or at least one structure (e.g., a building, a sign, a fire hydrant, etc.), etc. Each object 104 is stationary (e.g., located at a fixed location and over a period of time) or moves (e.g., has a speed and is associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations in area 108.

[0035] Routes 106a-106n (individually referred to as routes 106 and collectively referred to as routes 106) are each associated with (e.g., specify a series of actions (also referred to as trajectories) connecting states along which the AV can navigate. Each route 106 begins at an initial state (e.g., a state corresponding to a first spatiotemporal location and / or speed, etc.) and ends at a final target state (e.g., a state corresponding to a second spatiotemporal location different from the first spatiotemporal location) or a target zone (e.g., a subspace of acceptable states (e.g., terminal states)). In some embodiments, the first state includes a location where one or more individuals will board the AV, and the second state or zone includes one or more locations where one or more individuals boarding the AV will disembark. In some embodiments, routes 106 include multiple acceptable state sequences (e.g., multiple spatiotemporal location sequences) that are associated with (e.g., define multiple trajectories). In an example, routes 106 include only high-level actions or imprecise state locations, such as a series of connecting roads indicating a change of direction at a roadway intersection, etc. Additionally or alternatively, the route 106 may include more precise actions or states, such as, for example, specific target lanes or precise locations within lane regions and target speeds at those locations, etc. In an example, the route 106 includes a plurality of precise state sequences along at least one high-level action with a limited look-ahead horizon to an intermediate target, wherein a combination of consecutive iterations of the limited horizon state sequences cumulatively correspond to a plurality of trajectories that collectively form a high-level route terminating at a final target state or region.

[0036] The area 108 includes a physical area (e.g., a geographic region) in which the vehicle 102 can navigate. In an example, the area 108 includes at least one state (e.g., a country, a province, a separate state of a plurality of states included in a country, etc.), at least a portion of a state, at least one city, at least a portion of a city, etc. In some embodiments, the area 108 includes at least one named thoroughfare (referred to herein as a "road"), such as a highway, an interstate highway, a parkway, a city street, etc. Additionally or alternatively, in some examples, the area 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of an open space and / or undeveloped area, a dirt road, etc. In some embodiments, the road includes at least one lane (e.g., a portion of the road that the vehicle 102 can traverse). In an example, the road includes at least one lane associated with (e.g., identified based on) at least one lane marking line.

[0037] The vehicle-to-infrastructure (V2I) device 110 (sometimes referred to as a vehicle-to-infrastructure or vehicle-to-everything (V2X) device) includes at least one device configured to communicate with the vehicle 102 and / or the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, the fleet management system 116, and / or the V2I system 118 via the network 112. In some embodiments, the V2I device 110 includes a radio frequency identification (RFID) device, a sign, a camera (e.g., a two-dimensional (2D) and / or three-dimensional (3D) camera), lane markings, street lights, parking meters, etc. In some embodiments, the V2I device 110 is configured to communicate directly with the vehicle 102. Additionally or alternatively, in some embodiments, the V2I device 110 is configured to communicate with the vehicle 102, the remote AV system 114, and / or the fleet management system 116 via the V2I system 118. In some embodiments, the V2I device 110 is configured to communicate with the V2I system 118 via the network 112.

[0038] The network 112 includes one or more wired and / or wireless networks. In an example, the network 112 includes a cellular network (e.g., a long-term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of some or all of these networks, etc.

[0039] The remote AV system 114 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the network 112, the fleet management system 116, and / or the V2I system 118 via the network 112. In an example, the remote AV system 114 includes a server, a server group, and / or other similar devices. In some embodiments, the remote AV system 114 is co-located with the fleet management system 116. In some embodiments, the remote AV system 114 participates in the installation of some or all of the components of the vehicle (including autonomous systems, autonomous vehicle computing, and / or software implemented by autonomous vehicle computing, etc.). In some embodiments, the remote AV system 114 maintains (e.g., updates and / or replaces) these components and / or software during the life of the vehicle.

[0040] The queue management system 116 includes at least one device configured to communicate with the vehicles 102, the V2I devices 110, the remote AV system 114, and / or the V2I system 118. In an example, the queue management system 116 includes a server, a server group, and / or other similar devices. In some embodiments, the queue management system 116 is associated with a ridesharing company (e.g., an organization for controlling the operation of multiple vehicles (e.g., vehicles including autonomous systems and / or vehicles not including autonomous systems)).

[0041] In some embodiments, the V2I system 118 includes at least one device configured to communicate with the vehicle 102, the V2I device 110, the remote AV system 114, and / or the fleet management system 116 via the network 112. In some examples, the V2I system 118 is configured to communicate with the V2I device 110 via a connection other than the network 112. In some embodiments, the V2I system 118 includes a server, a server group, and / or other similar devices. In some embodiments, the V2I system 118 is associated with a municipality or a private agency (e.g., a private agency for maintaining the V2I device 110, etc.).

[0042] supply Figure 1 The number and arrangement of elements illustrated are examples. Figure 1 There may be additional elements, fewer elements, different elements, and / or differently arranged elements than those illustrated. Additionally or alternatively, at least one element of environment 100 may be described as being Figure 1 Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

[0043] Reference now Figure 2 , vehicle 200 (which can be Figure 1 102) includes, or is associated with, autonomous system 202, powertrain control system 204, steering control system 206, and braking system 208. In some embodiments, vehicle 200 is similar to vehicle 102 (see Figure 1). In some embodiments, the autonomous system 202 is configured to give the vehicle 200 autonomous driving capabilities (e.g., implementing at least one of the following driving automatic or maneuvering-based functions, features, and / or devices, etc., which enables the vehicle 200 to partially or completely operate without human intervention, including but not limited to fully autonomous vehicles (e.g., vehicles that abandon reliance on human intervention, such as Level 5 ADS-operated vehicles, etc.), highly autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in certain situations, such as Level 4 ADS-operated vehicles, etc.), and / or conditionally autonomous vehicles (e.g., vehicles that abandon reliance on human intervention in limited situations). In one embodiment, the autonomous system 202 includes the operational or tactical functions required to enable the vehicle 200 to operate in on-road traffic and continuously perform part or all of the dynamic driving tasks (DDT). In another embodiment, the autonomous system 202 includes an advanced driver assistance system (ADAS) including driver support features. The autonomous system 202 supports various levels of driving automation ranging from no driving automation (e.g., level 0) to full driving automation (e.g., level 5). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, reference may be made to SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, the entire contents of which are incorporated by reference. In some embodiments, the vehicle 200 is associated with an autonomous queue manager and / or a ride-sharing company.

[0044] Autonomous system 202 includes a sensor suite that includes one or more devices such as camera 202a, LiDAR sensor 202b, Radar sensor 202c, and microphone 202d. In some embodiments, autonomous system 202 may include more or fewer devices and / or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), and / or odometer sensors for generating data associated with an indication of the distance that vehicle 200 has traveled, etc.). In some embodiments, autonomous system 202 uses one or more devices included in autonomous system 202 to generate data associated with environment 100 described herein. Data generated by one or more devices of autonomous system 202 can be used by one or more systems described herein to observe the environment (e.g., environment 100) in which vehicle 200 is located. In some embodiments, autonomous system 202 includes communication device 202e, autonomous vehicle computing 202f, drive-by-wire (DBW) system 202h, and safety controller 202g.

[0045] The camera 202a includes a communication device 202e, an autonomous vehicle computer 202f, and / or a safety controller 202g configured to communicate with the communication device 202e via a bus (e.g., Figure 3 The camera 202a includes at least one device for communicating with the autonomous vehicle computing 202f (e.g., an image processing unit 202a, a bus ... Figure 1In some embodiments, the autonomous vehicle computing 202f determines a depth to one or more objects in a field of view of at least two of the plurality of cameras based on image data from the at least two cameras. In some embodiments, the camera 202a is configured to capture images of objects within a distance relative to the camera 202a (e.g., up to 100 meters and / or up to 1 kilometer, etc.). Thus, the camera 202a includes features such as sensors and lenses that are optimized for sensing objects at one or more distances relative to the camera 202a.

[0046] In an embodiment, the camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and / or other physical objects that provide visual navigation information. In some embodiments, the camera 202a generates traffic light data associated with the one or more images. In some examples, the camera 202a generates TLD (traffic light detection) data associated with one or more images including a format (e.g., RAW, JPEG, and / or PNG, etc.). In some embodiments, the camera 202a that generates TLD data differs from other systems incorporating cameras described herein in that the camera 202a may include one or more cameras with a wide field of view (e.g., a wide-angle lens, a fisheye lens, and / or a lens with a viewing angle of approximately 120 degrees or greater, etc.) to generate images related to as many physical objects as possible.

[0047] The light detection and ranging (LiDAR) sensor 202b includes a communication device 202e, an autonomous vehicle computing device 202f, and / or a safety controller 202g configured to communicate with the communication device 202e via a bus (e.g., Figure 3The LiDAR sensor 202b includes at least one device that communicates with a bus (the same or similar bus as the bus 302 of the embodiment of the present invention). The LiDAR sensor 202b includes a system configured to emit light from a light emitter (e.g., a laser emitter). The light emitted by the LiDAR sensor 202b includes light outside the visible spectrum (e.g., infrared light, etc.). In some embodiments, during operation, the light emitted by the LiDAR sensor 202b encounters a physical object (e.g., a vehicle) and is reflected back to the LiDAR sensor 202b. In some embodiments, the light emitted by the LiDAR sensor 202b does not penetrate the physical object encountered by the light. The LiDAR sensor 202b also includes at least one light detector that detects the light emitted from the light emitter after encountering the physical object. In some embodiments, at least one data processing system associated with the LiDAR sensor 202b generates an image (e.g., a point cloud and / or a combined point cloud, etc.) representing objects included in the field of view of the LiDAR sensor 202b. In some examples, at least one data processing system associated with the LiDAR sensor 202b generates an image representing the boundaries of the physical object and / or the surface of the physical object (e.g., the topology of the surface), etc. In such examples, the image is used to determine the boundaries of the physical object in the field of view of the LiDAR sensor 202b.

[0048] The radio detection and ranging (Radar) sensor 202c includes a sensor configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with a bus (same or similar bus as bus 302 of the embodiment of the present invention). Radar sensor 202c includes a system configured to transmit (pulsed or continuous) radio waves. The radio waves transmitted by Radar sensor 202c include radio waves within a predetermined spectrum. In some embodiments, during operation, the radio waves transmitted by Radar sensor 202c encounter physical objects and are reflected back to Radar sensor 202c. In some embodiments, the radio waves transmitted by Radar sensor 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with Radar sensor 202c generates a signal representing an object included in the field of view of Radar sensor 202c. For example, at least one data processing system associated with Radar sensor 202c generates an image representing the boundary of a physical object and / or the surface of a physical object (e.g., the topology of the surface), etc. In some examples, the image is used to determine the boundary of a physical object in the field of view of Radar sensor 202c.

[0049] The microphone 202d includes a microphone configured to communicate with the communication device 202e, the autonomous vehicle computing device 202f, and / or the safety controller 202g via a bus (e.g., Figure 3 At least one device for communicating with the vehicle 200 (the same or similar bus as bus 302 of FIG. 1 ). Microphone 202d includes one or more microphones (e.g., an array microphone and / or an external microphone, etc.) that capture an audio signal and generate data associated with (e.g., representing) the audio signal. In some examples, microphone 202d includes a transducer device and / or the like. In some embodiments, one or more systems described herein can receive the data generated by microphone 202d and determine the position (e.g., distance, etc.) of an object relative to vehicle 200 based on an audio signal associated with the data.

[0050] The communication device 202e includes at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. For example, the communication device 202e may include at least one device configured to communicate with the camera 202a, the LiDAR sensor 202b, the Radar sensor 202c, the microphone 202d, the autonomous vehicle computing 202f, the safety controller 202g, and / or the DBW (drive-by-wire) system 202h. Figure 3 The communication device 202e may be a device that is the same as or similar to the communication interface 314 of the vehicle. In some embodiments, the communication device 202e includes a vehicle-to-vehicle (V2V) communication device (eg, a device for enabling wireless communication of data between vehicles).

[0051] Autonomous vehicle computing 202f includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, safety controller 202g, and / or DBW system 202h. In some examples, autonomous vehicle computing 202f includes devices such as client devices, mobile devices (e.g., cellular phones and / or tablet computers, etc.), and / or servers (e.g., computing devices including one or more central processing units and / or graphics processing units, etc.). In some embodiments, autonomous vehicle computing 202f is the same or similar to autonomous vehicle computing 400 described herein. Additionally or alternatively, in some embodiments, autonomous vehicle computing 202f is configured to communicate with an autonomous vehicle system (e.g., with Figure 1 remote AV system 114 of the same or similar autonomous vehicle system), a fleet management system (e.g., Figure 1 of the same or similar queue management system as the queue management system 116), V2I devices (e.g., Figure 1 V2I device 110 that is the same as or similar to V2I device 110) and / or V2I system (e.g., Figure 1The V2I system 118 may communicate with the same or similar V2I system.

[0052] Safety controller 202g includes at least one device configured to communicate with camera 202a, LiDAR sensor 202b, Radar sensor 202c, microphone 202d, communication device 202e, autonomous vehicle computing 202f, and / or DBW system 202h. In some examples, safety controller 202g includes one or more controllers (electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). In some embodiments, safety controller 202g is configured to generate control signals that take precedence over (e.g., override) control signals generated and / or transmitted by autonomous vehicle computing 202f.

[0053] The DBW system 202h includes at least one device configured to communicate with the communication device 202e and / or the autonomous vehicle computing 202f. In some examples, the DBW system 202h includes one or more controllers (e.g., electrical controllers and / or electromechanical controllers, etc.) configured to generate and / or transmit control signals to operate one or more devices of the vehicle 200 (e.g., powertrain control system 204, steering control system 206, and / or braking system 208, etc.). Additionally or alternatively, one or more controllers of the DBW system 202h are configured to generate and / or transmit control signals to operate at least one different device of the vehicle 200 (e.g., turn signals, headlights, door locks, and / or windshield wipers, etc.).

[0054] The powertrain control system 204 includes at least one device configured to communicate with the DBW system 202h. In some examples, the powertrain control system 204 includes at least one controller and / or actuator, etc. In some embodiments, the powertrain control system 204 receives a control signal from the DBW system 202h, and the powertrain control system 204 causes the vehicle 200 to perform longitudinal vehicle movement (such as starting to move forward, stopping to move forward, starting to move backward, stopping to move backward, accelerating in a certain direction, decelerating in a certain direction, etc.), or to perform lateral vehicle movement (such as turning left and / or turning right, etc.). In an example, the powertrain control system 204 increases, maintains the same, or decreases the energy (e.g., fuel and / or electricity, etc.) provided to the motor of the vehicle, thereby rotating or not rotating at least one wheel of the vehicle 200.

[0055] The steering control system 206 includes at least one device configured to rotate one or more wheels of the vehicle 200. In some examples, the steering control system 206 includes at least one controller and / or actuator, etc. In some embodiments, the steering control system 206 rotates the two front wheels and / or the two rear wheels of the vehicle 200 to the left or right to turn the vehicle 200 left or right. In other words, the steering control system 206 causes the activity required for the regulation of the y-axis component of the vehicle's motion.

[0056] Braking system 208 includes at least one device configured to actuate one or more brakes to decelerate and / or hold vehicle 200 stationary. In some examples, braking system 208 includes at least one controller and / or actuator configured to cause one or more calipers associated with one or more wheels of vehicle 200 to close on respective rotors of vehicle 200. Additionally or alternatively, in some examples, braking system 208 includes an automatic emergency braking (AEB) system and / or a regenerative braking system, etc.

[0057] In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) for measuring or inferring a property of a state or condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, an inertial measurement unit (IMU), wheel rate sensors, wheel brake pressure sensors, wheel torque sensors, engine torque sensors, and / or steering angle sensors. Although brake system 208 is illustrated as being located at Figure 2 The braking system 208 is located on the proximal side of the vehicle 200 , but the braking system 208 can be located anywhere in the vehicle 200 .

[0058] Reference now Figure 3, a schematic diagram of an exemplary device 300. As illustrated, the device 300 includes a processor 304, a memory 306, a storage component 308, an input interface 310, an output interface 312, a communication interface 314, and a bus 302. In some embodiments, the device 300 corresponds to: at least one device of the vehicle 102 (e.g., at least one device of a system of the vehicle 102); at least one device of the V2I device 110; at least one device of the remote AV system 114; at least one device of the queue management system 116; at least one device of the V2I system 118; at least one device of the device 200 (e.g., at least one device of the autonomous system 202); at least one device of the DBW system 202h; at least one device of the powertrain control system 204; at least one device of the steering control system 206; and / or one or more devices of the network 112 (e.g., one or more devices of a system of the network 112). In some embodiments, one or more devices of vehicle 102 (e.g., one or more devices of a system of vehicle 102), at least one device of V2I device 110, at least one device of remote AV system 114, at least one device of queue management system 116, at least one device of V2I system 118, at least one device of device 200 (e.g., at least one device of autonomous system 202), at least one device of DBW system 202h, at least one device of powertrain control system 204, at least one device of steering control system 206, and / or one or more devices of network 112 (e.g., one or more devices of a system of network 112) include at least one device 300 and / or at least one component of device 300. Figure 3 As shown, apparatus 300 includes a bus 302 , a processor 304 , a memory 306 , a storage component 308 , an input interface 310 , an output interface 312 , and a communication interface 314 .

[0059] The bus 302 includes components that permit communication between components of the device 300. In some cases, the processor 304 includes a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or an accelerated processing unit (APU), etc.), a microphone, a digital signal processor (DSP), and / or any processing component that can be programmed to perform at least one function (e.g., a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), etc.). The memory 306 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic and / or static storage device (e.g., flash memory, magnetic memory, and / or optical memory, etc.) that stores data and / or instructions for use by the processor 304.

[0060] Storage component 308 stores data and / or software related to the operation and use of device 300. In some examples, storage component 308 includes a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk, etc.), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a tape, a CD-ROM, a RAM, a PROM, an EPROM, a FLASH-EPROM, an NV-RAM, and / or another type of computer-readable medium, and a corresponding drive.

[0061] The input interface 310 includes components that permit the device 300 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, and / or a camera, etc.). Additionally or alternatively, in some embodiments, the input interface 310 includes a sensor for sensing information (e.g., a global positioning system (GPS) receiver, an accelerometer, a gyroscope, and / or an actuator, etc.). The output interface 312 includes components for providing output information from the device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs), etc.).

[0062] In some embodiments, communication interface 314 includes a transceiver-like component (e.g., a transceiver and / or a separate receiver and transmitter, etc.) that permits device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. In some examples, communication interface 314 permits device 300 to receive information from another device and / or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface and / or cellular network interface, etc.

[0063] In some embodiments, the device 300 performs one or more processes described herein. The device 300 performs these processes based on the processor 304 executing software instructions stored by a computer-readable medium such as a memory 306 and / or a storage component 308. Computer-readable media (e.g., non-transitory computer-readable media) are defined herein as non-transitory memory devices. Non-transitory memory devices include storage space located within a single physical storage device or storage space distributed across multiple physical storage devices.

[0064] In some embodiments, the software instructions are read into the memory 306 and / or storage component 308 from another computer-readable medium or from another device via the communication interface 314. The software instructions stored in the memory 306 and / or storage component 308, when executed, cause the processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry is used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, unless expressly stated otherwise, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0065] The memory 306 and / or the storage component 308 include a data storage unit or at least one data structure (e.g., a database, etc.). The device 300 can receive information from the data storage unit or at least one data structure in the memory 306 or the storage component 308, store information in the data storage unit or at least one data structure, communicate information to the data storage unit or at least one data structure, or search for information stored in the data storage unit or at least one data structure. In some examples, the information includes network data, input data, output data, or any combination thereof.

[0066] In some embodiments, the device 300 is configured to execute software instructions stored in the memory 306 and / or the memory of another device (e.g., another device that is the same as or similar to the device 300). As used herein, the term "module" refers to at least one instruction stored in the memory 306 and / or the memory of another device, which, when executed by the processor 304 and / or the processor of another device (e.g., another device that is the same as or similar to the device 300), causes the device 300 (e.g., at least one component of the device 300) to perform one or more processes described herein. In some embodiments, the module is implemented in software, firmware, and / or hardware, etc.

[0067] supply Figure 3 The number and arrangement of components illustrated are examples. Figure 3 The device 300 may include additional components, fewer components, different components, or differently arranged components than the components illustrated. Additionally or alternatively, a set of components (e.g., one or more components) of the device 300 may perform one or more functions described as being performed by another component or another set of components of the device 300.

[0068] Reference now Figure 4, illustrates an example block diagram of an autonomous vehicle computing 400 (sometimes referred to as an "AV stack"). As illustrated, the autonomous vehicle computing 400 includes a perception system 402 (sometimes referred to as a perception module), a planning system 404 (sometimes referred to as a planning module), a positioning system 406 (sometimes referred to as a positioning module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in and / or implemented in an automatic navigation system of a vehicle (e.g., the autonomous vehicle computing 202f of the vehicle 200). Additionally or alternatively, in some embodiments, the perception system 402, the planning system 404, the positioning system 406, the control system 408, and the database 410 are included in one or more independent systems (e.g., one or more systems that are the same or similar to the autonomous vehicle computing 400, etc.). In some examples, the perception system 402, planning system 404, positioning system 406, control system 408, and database 410 are included in one or more independent systems located in the vehicle and / or at least one remote system as described herein. In some embodiments, any and / or all of the systems included in the autonomous vehicle computing 400 are implemented in software (e.g., software instructions stored in a memory), computer hardware (e.g., by a microprocessor, microcontroller, application specific integrated circuit (ASIC) and / or field programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. It will also be understood that in some embodiments, the autonomous vehicle computing 400 is configured to communicate with a remote system (e.g., an autonomous vehicle system that is the same or similar to the remote AV system 114, a fleet management system 116 that is the same or similar to the fleet management system 116, and / or a V2I system that is the same or similar to the V2I system 118, etc.).

[0069] In some embodiments, the perception system 402 receives data associated with at least one physical object in the environment (e.g., data used by the perception system 402 to detect at least one physical object) and classifies the at least one physical object. In some examples, the perception system 402 receives image data captured by at least one camera (e.g., camera 202a), the image being associated with (e.g., representing) one or more physical objects within the field of view of the at least one camera. In such examples, the perception system 402 classifies the at least one physical object based on one or more groups of physical objects (e.g., bicycles, vehicles, traffic signs, and / or pedestrians, etc.). In some embodiments, based on the classification of the physical object by the perception system 402, the perception system 402 transmits data associated with the classification of the physical object to the planning system 404.

[0070] In some embodiments, the planning system 404 receives data associated with a destination and generates data associated with at least one route (e.g., route 106) along which a vehicle (e.g., vehicle 102) can travel toward the destination. In some embodiments, the planning system 404 periodically or continuously receives data (e.g., data associated with the classification of physical objects described above) from the perception system 402, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the perception system 402. In other words, the planning system 404 can perform tasks related to tactical functions required to operate the vehicle 102 in traffic on the road. Tactical efforts involve maneuvering the vehicle in traffic during the journey, which includes but is not limited to deciding whether and when to overtake another vehicle, change lanes, or select an appropriate rate, acceleration, deceleration, etc. In some embodiments, the planning system 404 receives data associated with an updated position of the vehicle (e.g., vehicle 102) from the positioning system 406, and the planning system 404 updates at least one trajectory or generates at least one different trajectory based on the data generated by the positioning system 406.

[0071] In some embodiments, the positioning system 406 receives data associated with (e.g., representing) a location of a vehicle (e.g., vehicle 102) in an area. In some examples, the positioning system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensor 202b). In some examples, the positioning system 406 receives data associated with at least one point cloud from multiple LiDAR sensors, and the positioning system 406 generates a combined point cloud based on each point cloud. In these examples, the positioning system 406 compares the at least one point cloud or the combined point cloud with a two-dimensional (2D) and / or three-dimensional (3D) map of the area stored in the database 410. Then, based on the positioning system 406 comparing the at least one point cloud or the combined point cloud with the map, the positioning system 406 determines the position of the vehicle in the area. In some embodiments, the map includes a combined point cloud of the area generated before the navigation of the vehicle. In some embodiments, the map includes, but is not limited to, a high-precision map of roadway geometry, a map describing the road network connectivity, a map describing the physical properties of the roadway (such as traffic speed, traffic volume, the number of vehicle and bicycle traffic lanes, lane width, lane traffic direction or the type and location of lane markings, or a combination thereof, etc.), and a map describing the spatial location of road features (such as crosswalks, traffic signs, or various types of other driving lights, etc.). In some embodiments, the map is generated in real time based on data received by the perception system.

[0072] In another example, positioning system 406 receives global navigation satellite system (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, positioning system 406 receives GNSS data associated with a location of a vehicle in an area, and positioning system 406 determines the latitude and longitude of the vehicle in the area. In such an example, positioning system 406 determines the position of the vehicle in the area based on the latitude and longitude of the vehicle. In some embodiments, positioning system 406 generates data associated with the position of the vehicle. In some examples, based on the location of the vehicle determined by positioning system 406, positioning system 406 generates data associated with the position of the vehicle. In such an example, the data associated with the position of the vehicle include data associated with one or more semantic properties corresponding to the position of the vehicle.

[0073] In some embodiments, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle. In some examples, the control system 408 receives data associated with at least one trajectory from the planning system 404, and the control system 408 controls the operation of the vehicle by generating and transmitting control signals to operate the powertrain control system (e.g., DBW system 202h and / or powertrain control system 204, etc.), the steering control system (e.g., steering control system 206), and / or the braking system (e.g., braking system 208). For example, the control system 408 is configured to perform operational functions such as lateral vehicle motion control or longitudinal vehicle motion control. Lateral vehicle motion control causes the activity required for the regulation of the y-axis component of the vehicle motion. Longitudinal vehicle motion control causes the activity required for the regulation of the x-axis component of the vehicle motion. In the example, in the case where the trajectory includes a left turn, the control system 408 transmits a control signal to cause the steering control system 206 to adjust the steering angle of the vehicle 200, thereby causing the vehicle 200 to turn left. Additionally or alternatively, the control system 408 generates and transmits control signals to cause other devices of the vehicle 200 (eg, headlights, turn signals, door locks, and / or windshield wipers, etc.) to change states.

[0074] In some embodiments, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model (e.g., at least one multi-layer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, and / or at least one transformer, etc.). In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model alone or in combination with one or more of the above systems. In some examples, perception system 402, planning system 404, positioning system 406, and / or control system 408 implement at least one machine learning model as part of a pipeline (e.g., a pipeline for identifying one or more objects located in an environment, etc.).

[0075] Database 410 stores data transmitted to, received from, and / or updated by perception system 402, planning system 404, positioning system 406, and / or control system 408. In some examples, database 410 includes a storage component (e.g., a storage component) for storing data and / or software related to operations and using at least one system of autonomous vehicle computing 400. Figure 3In some embodiments, database 410 stores data associated with a 2D and / or 3D map of at least one area. In some examples, database 410 stores data associated with a 2D and / or 3D map of a portion of a city, portions of multiple cities, multiple cities, counties, states, and / or countries (State) (e.g., a country), etc. In such an example, a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200) can be driven along one or more drivable areas (e.g., single-lane roads, multi-lane roads, highways, remote roads, and / or off-road roads, etc.) and cause at least one LiDAR sensor (e.g., a LiDAR sensor that is the same or similar to LiDAR sensor 202b) to generate data associated with an image representing an object included in the field of view of the at least one LiDAR sensor.

[0076] In some embodiments, database 410 can be implemented across multiple devices. In some examples, database 410 includes a vehicle (e.g., a vehicle that is the same or similar to vehicle 102 and / or vehicle 200), an autonomous vehicle system (e.g., an autonomous vehicle system that is the same or similar to remote AV system 114), a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114), and a fleet management system (e.g., a vehicle that is the same or similar to remote AV system 114). Figure 1 The same or similar queue management system as the queue management system 116 of FIG. 1 and / or the V2I system (e.g., Figure 1 The V2I system 118 is the same as or similar to the V2I system 118).

[0077] In autonomous systems, such as autonomous vehicles (AVs), the positioning and / or orientation of a camera or camera can affect the near and long range field of view coverage. For example, most cameras are configured so that the horizontal field of view (HFoV) and the vertical field of view (VFoV) are different. Most commonly, the HFoV is wider than the VFoV (VFoV is tall), allowing more visibility in the horizontal direction and less visibility in the vertical direction. In other words, for many cameras, the entire field of view of the camera can be considered to be rectangular (longer in one direction than in another). In most cases, the camera is fixedly positioned, which means that the corresponding field of view of the camera is also fixed. However, as described herein, the ability to move, rotate, select and / or switch the positioning and / or orientation of the camera can allow different field coverage for both the horizontal field of view and the vertical field of view. For example, for a camera whose HFoV is typically larger than its VFoV, rotating the camera by approximately 90 degrees may cause the camera to capture an image with a narrower HFoV and a higher VFoV.

[0078] Rotating or otherwise moving the camera between different orientations may further adjust the shutter direction of the camera's rolling shutter. For example, for a camera that normally has a vertical rolling shutter orientation, rotating the camera 90 degrees will provide the camera with a horizontal rolling shutter orientation. As described herein, adjusting the position or orientation of a camera to adjust its rolling shutter orientation may be advantageous in many different situations, such as minimizing distortion in images of fast-moving objects and / or improving fusion of image data from the camera with other forms of data, such as LiDAR data.

[0079] Throughout the specification, the terms "orientation and position (positioning)" may be used interchangeably and generally refer to the orientation or position of a camera device, such as a camera. In the present specification, the term "normal" is used to refer to a camera device position or orientation in which the HFoV is greater than the VFoV and the rolling shutter direction of the camera device is vertical. The terms "horizontal" and "vertical" generally refer to the position relative to a vehicle including the camera system, but these terms are used only for ease of description and should not be construed as limiting. Throughout the specification, the terms "orientation at 90 degrees relative to normal" or "position (positioning) at 90 degrees relative to normal" may be used interchangeably. As described in more detail below, the orientation at 90 degrees relative to normal provides a horizontal rolling shutter for the camera system (e.g., the camera system has been rotated so that its normally vertical rolling shutter direction is now moved in a horizontal direction).

[0080] Figure 5A and Figure 5B are top and side views representing example field of view coverage of camera device 500 illustrated in a normal position or orientation, respectively. Figure 5A An example of a horizontal field of view 504 of the camera 500 is illustrated, and Figure 5B An example of a vertical field of view 508 of a camera 500 is illustrated, wherein the camera 500 is positioned in a normal orientation. The example angular ranges of the field of view shown in the figures are provided by way of example, and other cameras may be configured with fields of view of different ranges. As shown, the field of view of the camera 500 is asymmetric, wherein ( Figure 5A The HFoV shown is greater than ( Figure 5B In the case where the camera is provided in a normal orientation, the camera is oriented to have a vertical rolling shutter. This means that, with respect to the orientation of the drawings as shown, Figure 5B The shutter direction moves in the direction of arrow 512. For example, a vertical rolling shutter may capture image data from the top of the image to the bottom of the image, where the top of the image is captured slightly before the bottom of the image. Figure 5A The shutter direction is not shown in the figure, as the arrow will point into the page.

[0081] The horizontal field of view 504 and the vertical field of view 508 are asymmetrical. The horizontal field of view 504 is shown to provide 60 degrees of coverage, however, the degree of coverage can vary depending on the specific camera device 500. The vertical field of view 508 is shown to provide 39 degrees of coverage, however, the degree of coverage can vary depending on the specific camera device 500.

[0082] As described in more detail below, the camera device 500 can be rotated from a first position or orientation to a second position or orientation. The second position can be approximately 90 degrees relative to normal. When rotated to a position approximately 90 degrees relative to normal, the camera device 500 will be oriented such that it operates using a rolling shutter that moves in a horizontal direction. The direction of the horizontal rolling shutter can be orthogonal to the direction of the vertical rolling shutter. In this way, the shutter direction can be from left to right or from right to left. In some embodiments, the horizontal rolling shutter direction can be consistent with the direction of movement or rotation of the LiDAR device. When rotated to a position approximately 90 degrees relative to normal, the coverage of the horizontal field of view and the coverage of the vertical field of view will switch (e.g., the VFoV becomes greater than the HFoV). Using Figure 5A and Figure 5B As an example, with the camera moved to the second position (in this case, rotated to approximately 90 degrees relative to normal), the horizontal field of view will now have 39 degrees of coverage, while the vertical field of view will now have 60 degrees of coverage.

[0083] The ability to switch or select between a horizontal rolling shutter position and a vertical rolling shutter position by physically moving the camera device can allow different fields of view to be captured. Although there is a relationship between shutter direction and field of view, and a relationship between camera orientation and field of view, shutter direction and field of view are independent of each other, which means that a horizontal rolling shutter position does not necessarily require a specific field of view and a vertical rolling shutter position does not necessarily require a specific field of view. This can be beneficial because, as described herein, a horizontal rolling shutter and a vertical rolling shutter can provide different advantages.

[0084] Fig. 6A 516 (e.g., a vehicle) captured using a camera oriented with a vertical rolling shutter orientation. The shutter direction is represented by arrow 518. The motion of object 516 is represented by arrow 520. As shown, because object 516 is moving horizontally when the camera captures the image with a vertically oriented rolling shutter orientation, the shape of the object is distorted, e.g., appearing tilted as illustrated. This can be problematic for several reasons. On the one hand, the image may not accurately represent the real world scene. On the other hand, the distorted image data may not be consistent with data captured by other sensor modalities. For example, in Fig. 6A 5. LiDAR data (data captured by the LiDAR) is illustrated in dashed lines in FIG. 5. The LiDAR system can be configured to rotate with a rotation direction as shown by arrow 522. As illustrated, the LiDAR system captures undistorted data (represented by the dashed version of object 516). The solid outline of object 516 represents image data from a camera (e.g., camera 500). As shown, the use of a vertical rolling shutter can result in distortion between the image from the camera (e.g., solid outline) and the LiDAR data (e.g., dashed outline).

[0085] Figure 6B 5. Represents an image of object 516 captured using a camera oriented with a horizontal rolling shutter. The shutter direction is represented by arrow 524. The motion of object 516 is represented by arrow 526. The direction of rotation of the LiDAR is represented by arrow 530. The solid outline of object 516 represents an image from a camera (e.g., camera 500). The dashed outline represents LiDAR data. As shown, with the camera in this orientation, the image from the camera is more closely correlated to the LiDAR data, however they may or may not be the same. Fig. 6A and Figure 6B The scenario depicts an example situation where a horizontal rolling shutter can outperform a vertical rolling shutter.

[0086] Fig. 7A and Figure 7B Three camera devices 500a, 500b, 500c are shown positioned on (eg, mounted on the roof of) an AV 534. A LiDAR (Light Detection and Ranging) device 536 is also positioned on the AV 534. The LiDAR device 536 may include a rotational orientation 538 as shown. Fig. 7A shows the horizontal field of view, while Figure 7B The vertical field of view is shown.

[0087] In the illustrated configuration, the cameras 500a, 500b, 500c are positioned in a normal orientation. Although three cameras 500a, 500b, 500c are shown, any number of cameras 500 may be used. For example, one, two, three, or more than three cameras 500. Although the three cameras 500a, 500b, 500c shown are all positioned in a normal orientation, as discussed in more detail below, the cameras 500a, 500b, 500c may be rotated or otherwise moved independently of one another. For example, one camera 500a may be in a normal position, while two cameras 500b, 500c may be in a position 90 degrees relative to normal, or any possible combination of orientations. As described herein, the normal position may inherently have a vertical rolling shutter, while the position 90 degrees relative to normal may inherently have a horizontal rolling shutter.

[0088] The LiDAR device 536 may be configured to Fig. 7A The LiDAR device 536 can capture LiDAR images. As discussed above, the camera devices 500a, 500b, 500c are in a normal orientation, so the shutter direction is Figure 7B As shown by arrow 540 in FIG.

[0089] Each individual camera 500a, 500b, 500c may have a corresponding horizontal field of view 542a, 542b, 542c. As shown in sections 544a and 544b, the horizontal fields of view 542a, 542b, 542c may overlap at least partially. The overlap of the individual horizontal fields of view 542a, 542b, 542c may ensure more complete coverage, or a more complete data set is collected by the camera 500a, 500b, 500c. When the camera 500a, 500b, 500c is positioned in a normal orientation, there is a greater field of view overlap compared to the case where the camera 500a, 500b, 500c is positioned in an orientation 90 degrees relative to normal. In some embodiments, the normal orientation may provide a field of view overlap of about 1 degree overlap to about 59 degrees overlap (or any value between the two). However, no minimum overlap is required, and the overlap may depend on the camera or camera being used.

[0090] Each individual camera device 500a, 500b, 500c may have a corresponding vertical field of view. Figure 7B As shown, the corresponding vertical fields of view 546 can be substantially similar or identical in height.

[0091] Figure 7C and Fig.7D Shown with Fig. 7A and Figure 7BThe same AV 534 is shown, which includes camera devices 500a, 500b, 500c and LiDAR device 536. Figure 7C shows the horizontal field of view and Fig.7D shows the vertical field of view. Figure 7C and Fig.7D In the illustrated configuration, the camera device 500 has been rotated to an orientation of 90 degrees relative to normal.

[0092] Each individual camera 500a, 500b, 500c may have a corresponding horizontal field of view 542a, 542b, 542c. As shown in sections 544a and 544b, the horizontal fields of view 542a, 542b, 542c may at least partially overlap. Fig. 7A In comparison, Figure 7C In FIG. 5 , the overlapping horizontal field of view portions 544a, 544b are smaller. This is due to the positioning of the camera devices 500a, 500b, 500c. As described herein, the camera devices 500a, 500b, 500c have been moved from Fig. 7A The normal orientation in the Figure 7C 90 degrees relative to normal in the orientation. This results in less overlap of the horizontal field of view. This configuration can provide a field of view overlap of about 1 degree overlap to about 38 degrees overlap (or any value in between). The overlap can depend on the camera or camera being used, so the overlap can be outside the exemplary range if the camera or camera requires it.

[0093] In addition, the LiDAR device 536 can be configured to Figure 7C As mentioned, the camera devices 500a, 500b, 500c are oriented at 90 degrees relative to normal, so the shutter direction is Figure 7C As shown by arrow 548 in FIG. , the rolling shutter has also been transformed from a vertical rolling shutter to a horizontal rolling shutter. In this configuration, the shutter direction is substantially consistent with the rotation direction of the LiDAR system, which can facilitate correlation of the respective data sets as described above.

[0094] Each individual camera device 500a, 500b, 500c may have a corresponding vertical field of view. Fig.7D As shown, the corresponding vertical fields of view 546 can be substantially similar or identical in height. Fig.7D The vertical field of view 546 shown may be greater than Figure 7B Vertical field of view 546 shown in .

[0095] Fig. 8A and Figure 8BAn embodiment of a multi-position rolling shutter system 600 is shown. System 600 may include a camera 500, a connector 602, and an actuator 604. Camera 500 may, for example, be a camera assembly including one or more lenses and an image sensor. In some embodiments, camera 500 may be an image sensor. Connector 602 is configured to receive camera 500. For example, camera 500 may be fixedly connected to connector 602. Connector 602 may also be rotatably attached to actuator 604 so that the actuator may rotate camera 500 between two or more orientations. Actuator 604 may, for example, be an electric motor such as a stepper motor, although other actuators may also be used.

[0096] In some embodiments, system 600 may include at least one processor and at least one non-transitory storage medium. The non-transitory storage medium may store instructions that, when executed by the processor, may cause the processor to cause actuator 604 to rotate camera device 500. As described herein, system 600 may be used in conjunction with AV and LiDAR devices. AV may utilize more than one system 600 at a time. For example, as described in reference FIG. 7A to FIG. 7D As shown and described, multiple systems 600 may be attached to the AV.

[0097] Fig. 8A Camera device 500 is shown in a normal orientation. Figure 8B Camera 500 is shown in an orientation of 90 degrees relative to normal. As described herein, camera 500 can be configured to capture images within a field of view using a rolling shutter that operates in a shutter direction. By rotating camera 500 using actuator 604, the field of view and shutter direction can be changed.

[0098] Connector 602 can rotatably couple camera device 500 to actuator 604. Connector 602 can be coupled to actuator 604 at a first end. Connector 602 can be coupled to camera device 500 at a second end. Fig. 8A and Figure 8B As shown, connector 602 can be coupled to camera device 500 at two connection points. Connector 602 can be coupled to camera device 500 at a single connection point. Connector 602 can have a generally U-shape. Connector 602 can be, for example, a single segment or rod having a single connection point with camera device 500 and a single connection point with actuator 604. Connector 602 can be configured to rotate with camera device 500. Connector 602 can be configured to be stationary relative to camera device 500.

[0099] The actuator 604 can be configured to rotate the connector 602 and the camera device 500. In some embodiments, the actuator 604 can be a motor. Figure 8C Starting at block 608, an image may be received from camera 500 using at least one processor. Proceeding to block 610, the image may be analyzed using the processor to determine at least one image parameter. Proceeding to block 612, the camera may be rotated in response to the analysis performed at 610. For example, actuator 604 may rotate camera 500 and / or connector 602 between a first orientation and a second orientation. For example, the first orientation may be such as Fig. 8A As shown and the second direction is as Figure 8B As shown. However, the first orientation can be any orientation, for example, the normal orientation as shown, the orientation at 90 degrees relative to normal, or the orientation at any degree relative to normal. In addition, the second orientation can be any orientation, for example, the orientation at 90 degrees relative to normal as shown, the normal orientation, or the orientation at any degree relative to normal. The image being analyzed can be used to infer the speed of the object in the image plane by analyzing the motion blur and skew of the object. This information can then be used in the overall decision of whether to trigger rotation.

[0100] The first orientation may have a shutter orientation in a first direction. The second orientation may have a shutter orientation in a second direction. The first direction may be different from the second direction. For example, the first direction may be a vertical rolling shutter and the second direction may be a horizontal rolling shutter, or the first direction may be a horizontal rolling shutter and the second direction may be a vertical rolling shutter.

[0101] In some embodiments, the rotation of the camera 500 can be based at least in part on a predetermined threshold or benchmark, or the rotation of the camera 500 can be triggered based at least in part on a predetermined threshold or benchmark. For example, the rotation of the camera 500 can be responsive to a confidence level, a comparison between an image captured from the camera and the LiDAR data, the height and / or width of a detected object, and whether the ground is visible. The decision to rotate the camera 500 can be automatic. In some embodiments, the decision to rotate the camera 500 can be made by a user (e.g., by a driver). In some embodiments, the decision to rotate the camera 500 can be made with the assistance of a remote vehicle.

[0102] In some embodiments, rotation can be triggered based on the calculation of the intersection-over-union (IOU) between objects in the camera image and the corresponding LiDAR points projected onto the image. If the IOU is close to 1, then the correlation indicates that rotation is not needed. If the IOU is less than 1 or begins to drop, rotation of the camera or imaging device can be triggered to try to improve the IOU.

[0103] In some embodiments, the processor may be configured to detect an object using a LiDAR image. The processor may cause the actuator 604 to rotate the camera 500 between a first orientation and a second orientation based on determining the difference between the shape of the object detected in the image from the camera 500 and the shape of the object detected in the LiDAR image. The rotation may be triggered by the difference exceeding a predetermined threshold. For example, by considering the IOU as described above.

[0104] In some embodiments, the processor may be configured to cause the actuator 604 to rotate the camera device 500 between the first orientation and the second orientation based on a determination that the object extends beyond the field of view of the camera device 500 .

[0105] Other examples that may trigger the rotation of the camera 500 include the detection of data differences between the camera 500 and the LiDAR. In this example, the camera may be rotated to see if the data correlation improves. This may be triggered when a high speed object is detected.

[0106] Other situations include when the object to be detected is close to the vehicle and is moving quickly. It may be necessary to enable all cameras in a horizontal rolling shutter orientation to fully capture the object. Examples of situations that may trigger this include vehicles cutting into or crossing traffic.

[0107] An example situation where a vertical rolling shutter would be beneficial is when there is a close, slow moving object that would simultaneously occupy the overlapping area of ​​both cameras 500. A vertical rolling shutter would allow for better image stitching.

[0108] Fig. 9A and Fig. 9B The vertical field of view is shown, and examples are provided regarding when it may be necessary to rotate a camera (eg, camera 500 ) from a normal orientation to an orientation that is 90 degrees relative to normal. Fig. 9A The vertical field of view is shown representing the camera in a normal orientation. Fig. 9B Shown is a representation of the vertical field of view of the camera at an orientation of 90 degrees relative to normal.

[0109] like Fig. 9A As shown, AV 620 is approaching traffic light 624. The vertical field of view of camera 500 does not reach the top of traffic light 624. In this example, camera 500 can be rotated to an orientation of 90 degrees relative to normal as described herein. Fig. 9BAs shown, camera 500 has been rotated to an orientation of 90 degrees relative to normal. The vertical field of view now reaches the top of traffic light 624. This is an example of a situation in which it may be necessary to rotate the positioning of camera 500. Although the depicted scenario illustrates the need to vary the vertical field of view, similar situations may arise in which it may be necessary to rotate camera 500 to allow for a wider or narrower horizontal field of view.

[0110] As described herein, the normal orientation and the orientation at 90 degrees relative to normal provide various advantages and benefits. The adjustability of the field of view (both vertical and horizontal) provides many advantages. For example, the normal orientation provides more horizontal field of view overlap between multiple cameras, which can allow better stitching of images. For example, the orientation at 90 degrees relative to normal provides a higher vertical field of view, which can allow better coverage of taller objects.

[0111] Adjustability of the orientation of the rolling shutter (vertical and horizontal) also provides a number of advantages. For example, a vertical rolling shutter allows the edges of the captured images to be closer in time for easier stitching. This is because the same rows in both images are exposed at the same time. A horizontal rolling shutter allows for better overlap with LiDAR data, for example.

[0112] In the previous description, aspects and embodiments of the present disclosure have been described with reference to many specific details, which may vary depending on the implementation. Therefore, the description and the accompanying drawings should be regarded as illustrative, not restrictive. The only and exclusive indication of the scope of the invention, and the applicant's expectation that the content of the scope of the invention is the literal and equivalent scope of the claims issued from this application in the specific form of the claims, including any subsequent amendments. Any definition of the terms used to be included in such claims that are clearly set forth herein should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the attached claims, the following of the phrase may be an additional step or entity, or a sub-step / sub-entity of the previously described step or entity.

[0113] Various additional example embodiments of the present disclosure may be described by the following clauses:

[0114] Clause 1: A system comprising:

[0115] a camera device configured to capture an image within a field of view using a rolling shutter operated in a shutter direction;

[0116] a connector coupled to the camera device;

[0117] An actuator coupled to the connector, the actuator being configured to rotate the connector and the camera device between the following two orientations:

[0118] a first orientation, wherein the shutter direction comprises a first direction, and

[0119] a second orientation, wherein the shutter direction comprises a second direction different from the first direction;

[0120] at least one processor; and

[0121] At least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to cause the actuator to rotate the camera device between the first orientation and the second orientation based on an analysis of the image.

[0122] Clause 2: The system of clause 1, wherein the field of view of the camera device is asymmetric.

[0123] Clause 3: The system of clause 1 or 2, wherein the first direction is orthogonal to the second direction.

[0124] Clause 4: The system of clause 3, wherein the first direction is a vertical direction and the second direction is a horizontal direction.

[0125] Clause 5: A system according to any one of clauses 1 to 4, further comprising a light detection and ranging system (LiDAR system), the LiDAR system being configured to capture LiDAR images, and wherein the at least one processor is configured to:

[0126] detecting an object within an image from the camera;

[0127] detecting the object within the LiDAR image; and

[0128] The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that a difference between a shape of an object detected in an image from the camera and a shape of an object detected in the LiDAR image exceeds a threshold.

[0129] Clause 6: A system according to any one of clauses 1 to 5, wherein the at least one processor is configured to:

[0130] detecting an object within the image; and

[0131] The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that the object extends beyond the field of view of the camera.

[0132] Clause 7: The system of any one of clauses 1 to 6, wherein the rotation is responsive to a predetermined threshold.

[0133] Clause 8: The system of clause 7, wherein the predetermined threshold is a confidence level.

[0134] Clause 9: A method comprising:

[0135] Using at least one processor, receiving an image from a camera device configured to capture an image within a field of view using a rolling shutter operated in a shutter direction, wherein the camera device is coupled to an actuator configured to move the camera device between a first orientation in which the shutter direction comprises a first direction and a second orientation in which the shutter direction comprises a second direction different from the first direction;

[0136] using the at least one processor, analyzing the image to determine at least one image parameter; and

[0137] Using the at least one processor, the actuator is caused to rotate the camera between the first orientation and the second orientation based on the at least one image parameter.

[0138] Clause 10: The method of clause 9, wherein rotating the camera is based at least in part on a predetermined threshold.

[0139] Clause 11: The method of clause 10, wherein the predetermined threshold is a confidence level.

[0140] Clause 12: The method according to any one of clauses 9 to 11, further comprising:

[0141] detecting an object within an image received from the camera;

[0142] detecting the object within a LiDAR image received from a LiDAR system; and

[0143] The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that a difference between a shape of an object detected in an image from the camera and a shape of an object detected in the LiDAR image exceeds a threshold.

[0144] Clause 13: The method according to any one of clauses 9 to 12, further comprising:

[0145] detecting an object within the image; and

[0146] The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that the object extends beyond the field of view of the camera.

[0147] Clause 14: A device comprising:

[0148] an imaging device comprising an asymmetric field of view including a first dimension that is smaller than and orthogonal to a second dimension, the imaging device being configured to capture an image within the field of view using a rolling shutter that operates in a shutter direction parallel to the first dimension;

[0149] a connector coupled to the camera device; and

[0150] An actuator coupled to the connector, the actuator being configured to rotate the connector and the camera device between the following two orientations:

[0151] a first orientation, wherein the first dimension is parallel to the first direction, and

[0152] A second orientation, wherein the first dimension is parallel to a second direction orthogonal to the first direction.

[0153] Clause 15: The apparatus of clause 14, wherein the first orientation differs from the second orientation by approximately 90 degrees.

[0154] Clause 16: The apparatus of clause 14 or 15, wherein:

[0155] The first direction is a vertical direction, and the second direction is a horizontal direction,

[0156] In the first orientation, the shutter direction is parallel to the vertical direction, and

[0157] In the second orientation, the shutter direction is parallel to the horizontal direction.

[0158] Clause 17: Apparatus as recited in any one of clauses 14 to 16, wherein the imaging device comprises a camera.

[0159] Clause 18: The apparatus of any of clauses 14 to 17, wherein the actuator comprises an electric motor.

[0160] Clause 19: Apparatus according to any one of clauses 14 to 18, wherein the rotation is responsive to a predetermined threshold.

[0161] Clause 20: The apparatus of clause 19, wherein the predetermined threshold is a confidence level.

Claims

1. A system, include: a camera device configured to capture an image within a field of view using a rolling shutter operated in a shutter direction; a connector coupled to the camera device; An actuator coupled to the connector, the actuator being configured to rotate the connector and the camera device between the following two orientations: a first orientation, wherein the shutter direction comprises a first direction, and a second orientation, wherein the shutter orientation comprises a second orientation different from the first orientation; at least one processor; as well as At least one non-transitory storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to cause the actuator to rotate the camera device between the first orientation and the second orientation based on an analysis of the image.

2. The system according to claim 1, in, The field of view of the camera device is asymmetric.

3. The system according to claim 1 or 2, in, The first direction is orthogonal to the second direction.

4. The system according to claim 3, in, The first direction is a vertical direction, and the second direction is a horizontal direction.

5. The system according to any one of claims 1 to 4, further comprising a light detection and ranging system (LiDAR system), the LiDAR system being configured to capture LiDAR images, and in, The at least one processor is configured to: detecting an object within an image from the camera; detecting the object within the LiDAR image; as well as The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that a difference between a shape of an object detected in an image from the camera and a shape of an object detected in the LiDAR image exceeds a threshold.

6. The system according to any one of claims 1 to 5, in, The at least one processor is configured to: detecting an object within the image; and The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that the object extends beyond the field of view of the camera.

7. The system according to any one of claims 1 to 6, in, The rotation is responsive to a predetermined threshold.

8. The system according to claim 7, in, The predetermined threshold is a confidence level.

9. A method, include: Using at least one processor, receiving an image from a camera device configured to capture an image within a field of view using a rolling shutter operated in a shutter direction, wherein the camera device is coupled to an actuator configured to move the camera device between a first orientation in which the shutter direction comprises a first direction and a second orientation in which the shutter direction comprises a second direction different from the first direction; analyzing, using the at least one processor, the image to determine at least one image parameter; as well as Using the at least one processor, the actuator is caused to rotate the camera between the first orientation and the second orientation based on the at least one image parameter.

10. The method according to claim 9, in, Rotating the camera is based at least in part on a predetermined threshold.

11. The method according to claim 10, in, The predetermined threshold is a confidence level.

12. The method according to any one of claims 9 to 11, further comprising: include: detecting an object within an image received from the camera; detecting the object within a LiDAR image received from a LiDAR system; as well as The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that a difference between a shape of an object detected in an image from the camera and a shape of an object detected in the LiDAR image exceeds a threshold.

13. The method according to any one of claims 9 to 12, further comprising: include: detecting an object within the image; as well as The actuator is caused to rotate the camera between the first orientation and the second orientation based on a determination that the object extends beyond the field of view of the camera.

14. A device, include: an imaging device comprising an asymmetric field of view including a first dimension that is smaller than and orthogonal to a second dimension, the imaging device being configured to capture an image within the field of view using a rolling shutter that operates in a shutter direction parallel to the first dimension; a connector coupled to the camera device; as well as An actuator coupled to the connector, the actuator being configured to rotate the connector and the camera device between the following two orientations: a first orientation, wherein the first dimension is parallel to the first direction, and A second orientation, wherein the first dimension is parallel to a second direction orthogonal to the first direction.

15. The device according to claim 14, in, The first orientation differs from the second orientation by approximately 90 degrees.

16. The device according to claim 14 or 15, in, The first direction is a vertical direction, and the second direction is a horizontal direction, In the first orientation, the shutter direction is parallel to the vertical direction, and In the second orientation, the shutter direction is parallel to the horizontal direction.

17. The apparatus according to any one of claims 14 to 16, in, The imaging device includes a camera.

18. The apparatus according to any one of claims 14 to 17, in, The actuator includes an electric motor.

19. The apparatus according to any one of claims 14 to 18, in, The rotation is responsive to a predetermined threshold.

20. The device according to claim 19, in, The predetermined threshold is a confidence level.

Images