Method of augmenting an environmental scene and autonomous vehicle testing system

Abstract

A method for enhancing an environmental scene includes: acquiring an image from an autonomous vehicle, the image being captured by a camera mounted on the autonomous vehicle and depicting the environment surrounding the autonomous vehicle; generating a virtual object graph containing one or more virtual objects, wherein rendering the virtual object graph on the image produces an object-enhanced image; generating a global scene graph characterizing a macroscopic static effect of weather; generating a weather dynamic effect graph representing at least one specific weather dynamic effect; synthesizing an environment-enhanced image based on the virtual object graph, the global weather scene graph, and the weather dynamic effect graph, such that the visual representation of the environment is as it would behave under predetermined weather and traffic conditions; and inputting the synthesized environment-enhanced image into an onboard vehicle controller of the autonomous vehicle, causing the autonomous vehicle to perform at least one autonomous driving operation based on the environment-enhanced image.

Description

Technical Field

[0001] The present invention relates to a method and system for rendering a scene used by an autonomous vehicle during testing. Background Technology

[0002] There are no existing methods or systems for using weather-related and traffic-related graphics to enhance images captured by autonomous vehicles (AVs). Summary of the Invention

[0003] A method for enhancing an environmental scene for use by an autonomous vehicle during testing includes the following steps: obtaining an image representing scene information from the autonomous vehicle, wherein the image is captured by a camera mounted on the autonomous vehicle and depicts the environment in which the autonomous vehicle is driving; generating virtual object graphics representing one or more virtual objects, wherein an object-enhanced image is generated when the virtual object graphics are rendered on the image; generating a global scene graphic representing a macroscopic static effect of weather to simulate the macroscopic visual effect of the environment under predetermined weather conditions based on the object-enhanced image, wherein the global scene graphic is generated by one or more artificial intelligence (AI) techniques; generating a weather dynamic effect graphic representing at least one specific weather dynamic effect, wherein the weather dynamic effect graphic is generated by one or more physically based computational techniques; generating a synthetic weather object-enhanced image based on the virtual object graphics, the global scene graphic, and the weather dynamic effect graphic; and inputting the synthetic weather object-enhanced image into an onboard vehicle controller of the autonomous vehicle, causing the autonomous vehicle to perform at least one autonomous driving operation based on the synthetic weather object-enhanced image. Attached Figure Description

[0004] Preferred exemplary embodiments will now be described with reference to the accompanying drawings, wherein similar reference numerals denote similar elements, and wherein:

[0005] According to the first embodiment, Figure 1 A communication system comprising an autonomous vehicle (AV) and a test site server is described, which can be used to perform one or more of the methods described herein;

[0006] According to the second embodiment, Figure 2 A communication system comprising an autonomous vehicle (AV) and a test site server is described, which can be used to perform one or more of the methods described herein;

[0007] Figure 3 This is a flowchart of an enhanced environmental scenario method according to one embodiment, used for testing autonomous vehicles;

[0008] Figure 4This is a flowchart of an enhanced environmental scenario method according to one embodiment, for use by autonomous vehicles during testing of autonomous vehicles;

[0009] Figure 5 This is a flowchart of a process for generating and rendering virtual object graphics according to one embodiment, which can be used as... Figure 3 and Figure 4 Part of the method;

[0010] Figure 6 This is a flowchart of a process for generating weather-related graphics according to one embodiment, which can be used as... Figure 3 and Figure 4 It is part of the method. Detailed Implementation

[0011] The systems and methods described herein enable the augmentation of images captured by an autonomous vehicle (AV) using weather-related and traffic-related graphics, such that the augmented image displays a realistic visual representation of the environment as it would behave under predetermined or selected weather and / or other environmental conditions. Furthermore, the systems and methods described herein enable the augmented image to be input into the AV's onboard vehicle controller, causing the AV to perform at least one autonomous driving operation based on the augmented image. This allows the AV to be tested under a variety of different weather conditions, such as rain or snow. In one embodiment, virtual object graphics representing one or more virtual objects, such as virtual vehicles or pedestrians (or other traffic-related objects), are also generated and then combined with weather-related graphics to form a synthetic augmented image that displays a realistic visual representation of the environment as it would behave under predetermined weather conditions, and as if the virtual objects were real objects present in the environment. This allows the AV to be tested according to various combinations of traffic scenarios / conditions and weather conditions.

[0012] According to some embodiments, weather-related graphics can be generated in two steps. First, a global scene graph is generated, and when rendered together with an object-enhanced image, the global scene graph produces a visual representation of the environment as it would behave under predetermined weather conditions. Second, a weather dynamics graph representing at least one specific weather dynamic effect is generated. As an example, if the predetermined weather condition is snowing, the global scene graph, when rendered together with virtual object graphics or other images / graphics, produces a visual representation of a snow-covered environment (e.g., snow covering roads, snow covering virtual vehicles or other virtual objects, snow covering real vehicles or other real objects). In the same example, if the predetermined weather condition is snowing, the weather dynamics graph, when rendered together with an image, causes snowflakes to appear in the camera's view. For example, when the camera's field of view is through the windshield, snowflakes can be rendered as if they were outside the windshield and flying in the sky. In one example, if the predetermined weather condition is raining, the weather dynamics graph can represent raindrops appearing on and / or flowing down the windshield. In one embodiment, one or more artificial intelligence (AI) technologies are used to generate global scene graphics, and one or more physics-based computational techniques are used to generate dynamic weather effect graphics.

[0013] like Figure 1 As shown, the operating environment includes a communication system 10 (for test site 12), a test site server 14, an autonomous vehicle (AV) 16 with vehicle electronics 18, a terrestrial network 20, a wireless carrier system 22, and a Global Navigation Satellite System (GNSS) satellite constellation 24. Test site 12 is an example of a vehicle testing environment in which one or more methods described herein can be performed or used. In some embodiments, the vehicle testing environment may be located at a private test site, while in other embodiments, the vehicle testing environment may include one or more public roads or areas, such as parking lots. It should be understood that, although Figure 1 The illustrated embodiment provides an example of such a communication system 10, but the systems and methods described below can be used as part of a variety of other communication systems.

[0014] Terrestrial network 20 may be a conventional terrestrial communication network connected to one or more terrestrial lines, and connects wireless carrier system 22 to test site server 14. For example, terrestrial network 20 may include a Public Switched Telephone Network (PSTN) that can be used to provide hard-wired telephone, packet-switched data communications, and Internet infrastructure. One or more segments of terrestrial network 20 may be implemented through standard wired networks, fiber optic or other optical networks, cable networks, power lines, other wireless networks such as wireless local area networks (WLANs), or networks providing broadband wireless access (BWA), or any combination thereof.

[0015] The wireless carrier system 22 can be any suitable remote data transmission system, such as a cellular telephone system. The wireless carrier system 22 is shown as including a single cell tower 26; however, depending on the cellular technology being used, the wireless carrier system 22 may include additional cell towers and one or more of the following components: base transceiver stations, mobile switching centers, base station controllers, evolved Nodes (e.g., eNodeBs), mobility management entities (MMEs), serving and PGN gateways, etc., and any other network components for connecting the wireless carrier system 22 to the terrestrial network 20 or to user equipment (UEs) (e.g., may include a telematics device in AV16), all of which are generally indicated in 28. The wireless carrier system 22 can implement any suitable communication technology, including GSM / GPRS, CDMA or CDMA2000, LTE, 5G, etc. In at least one embodiment, the wireless carrier system 22 implements 5G cellular communication technology and includes suitable hardware and configuration. In some such embodiments, the wireless carrier system 22 provides a 5G network usable by AV 16 for communicating with the test site server 14 or other computers / devices located remotely from AV 16. Typically, the wireless carrier system 22, its components, the arrangement of its components, the interactions between the components, etc., are well known in the art.

[0016] Test site server 14 can be used to provide a backend for one or more components of test site 12. In at least one embodiment, test site server 14 includes one or more computers or computing devices (collectively, "computers") configured to perform one or more steps of the methods described herein. In another embodiment, test site server 14 is used to store information about one or more components of test site 14 and / or information belonging to AV 16, such as vehicle status information that can be used to evaluate the performance of AV 16 during testing. Test site server 14 is a server executed or hosted by one or more computers, each computer including a processor and non-transitory computer-readable storage accessible by the processor.

[0017] In the illustrated embodiment, AV 16 is depicted as a passenger vehicle, but it should be understood that any other vehicle may be used, including motorcycles, trucks, sports utility vehicles (SUVs), recreational vehicles (RVs), bicycles, other vehicles or mobile devices that can be used on roads or sidewalks, etc. As depicted in the illustrated embodiment, AV 16 includes vehicle electronics 18, which includes an onboard vehicle computer 30, a GNSS receiver 32, one or more cameras 34, a lidar sensor 36, and a vehicle communication bus 38. Figure 1 Examples of specific components of vehicle electronics 18 are provided; however, it should be understood that, according to various embodiments, in addition to Figure 1 In addition to the components depicted, vehicle electronics 18 may include one or more other components to supplement or replace them. Figure 1 Components.

[0018] Global Navigation Satellite System (GNSS) receiver 32 receives radio signals from GNSS satellite constellation 24. GNSS receiver 32 uses the received radio signals to generate location data representing the location of GNSS receiver 32 and, therefore, the location of AV 16 where GNSS receiver 32 is installed. In one embodiment, for example, if test site 12 is located in the United States, GNSS receiver 32 may be a Global Positioning System (GPS) receiver. In another embodiment, for example, if test site 12 is located in Europe, GNSS receiver 32 may be a GNSS receiver configured for use with Galileo (a satellite navigation system). In addition to location data that can represent locations as pairs of geographic coordinates, GNSS receiver 32 may also specify the time associated with each location. This time and location data obtained by the GNSS receiver from GNSS signals is called GNSS data.

[0019] Each of the one or more cameras 34 is used to acquire an image of the vehicle environment, and the image captured by camera 34 can be represented as a pixel array specifying color information. Each camera 34 can be any suitable digital camera or image sensor, such as a complementary metal-oxide-semiconductor (CMOS) camera / sensor. Each camera 34 is connected to the vehicle communication bus 38 and can provide images to the onboard vehicle computer 30. In some embodiments, images from one or more cameras 34 are provided to a test site server 14. At least one of the cameras 34 is mounted on the AV 16 such that the field of view of the at least one camera is pointed towards the external environment of the vehicle. In at least some embodiments, the image captured by the at least one camera represents scene information, and this scene information depicts the external environment of the AV 16.

[0020] Images captured by camera 34 may include visual descriptions of various external objects, such as one or more roads (e.g., the road on which AV 16 travels), one or more pedestrians, one or more other vehicles, buildings, traffic signs, other roadside infrastructure, one or more trees or other living things, traffic signals (e.g., traffic lights), clouds, sky, etc. One or more cameras 34 may be any one of the following: a front camera mounted in front of AV 16 and facing the front area of ​​AV 16; a side camera mounted on the side of AV 16 and facing the side area of ​​AV 16; or a rear camera mounted on the back or rear of AV 16 and facing the rear area of ​​AV 16.

[0021] The lidar sensor 36 is used to acquire lidar sensor data of one or more objects in the environment, and the lidar sensor data may include range and / or location information of these objects. Non-visible light waves are emitted by the lidar sensor 36, reflected by the objects, and then captured by the lidar sensor 36. It should be understood that various types of lidar devices can be used, including, for example, those from Velodyne. TM The manufactured equipment, such as Alpha Prime TM Ultra Puck TM Puck TM The lidar sensor 36 is connected to the vehicle communication bus 38 and can provide lidar sensor data to the on-board vehicle computer 30. Although only a single lidar sensor is shown and described herein, it should be understood that, according to at least some embodiments, the AV 16 may include two or more lidar sensors.

[0022] The vehicle computer 30 is an on-board computer because it is carried by AV 16, and is considered a vehicle computer because it is part of vehicle electronics 18. The vehicle computer 30 includes a processor 40 and a non-transitory computer-readable storage 42 accessible by the processor 40. The vehicle computer 30 can be used for various processes executed at AV 16, and in at least one embodiment, for performing one or more steps of one or more methods described herein. The vehicle computer 30 is connected to a vehicle communication bus 38 and can use the bus 38 to send messages to and receive messages from other vehicle components. The vehicle computer 30 also includes a short-range wireless communication (SRWC) line 44 for wireless communication and a cellular chipset 46. The SRWC line 44 includes an antenna and is configured to perform one or more SRWC technologies, such as the IEEE 802.11 protocol (e.g., IEEE 802.11p, Wi-Fi). TM WiMAX TM ZigBee TM Z-Wave TM Wi-Fidirect TM ,Bluetooth TM (For example, Bluetooth) TMThe SRWC line 44 can be used to perform communication with the test site server 14—for example, AV 16 can use the SRWC line 44 to send a message to a roadside device (RSE) (not shown), which can then forward the message to the test site server 14 via the terrestrial network 20 to which the RSE is connected. The cellular chipset 46 includes an antenna and is used to perform cellular or long-range radio communication with the radio carrier system 22. Furthermore, in one embodiment, the cellular chipset 46 includes adapted 5G hardware and a 5G configuration to enable 5G communication between AV 16 and the radio carrier system 22, such as communication between AV 16 and one or more remote devices / computers, such as those representing the test site server 14.

[0023] In one embodiment, AV 16 is or includes an onboard vehicle controller for performing autonomous driving operations, and in this sense, may be referred to as an autonomous driving controller. In other embodiments, the autonomous driving controller may be independent of the onboard vehicle computer 30, but may be coupled to the onboard vehicle computer 30 directly or via a communication bus 38 or other suitable communication network. The autonomous driving controller can be used to make specific decisions regarding the autonomous driving operations of AV 16, such as whether to perform maneuvers, apply braking, accelerate the vehicle, etc.

[0024] The test site server 14 illustrated includes one or more processors 48 and non-transitory computer-readable storage 50. In one embodiment, the test site server is used to perform one or more steps of one or more methods described herein, such as method 200 and / or method 300 discussed below. In such an embodiment, the test site server 14 may be configured such that when computer instructions stored on the memory 50 are executed by the processor 48, the test site server 14 causes specific steps and / or functions, such as any functions attributed to the test site server 14 discussed herein, to be performed. In one embodiment, the processor 48 and the memory 50 storing computer instructions may form an autonomous vehicle test system configured to perform one or more steps of one or more methods described below. In such an embodiment, the autonomous vehicle test system is formed by remote components of AV 16, and the system may be referred to as a remotely based autonomous vehicle test system. In one embodiment, at least one of the one or more processors 48 is a graphics processing unit (GPU).

[0025] In another embodiment, one or more processors mounted on AV 16 and memory mounted on AV 16 form an autonomous vehicle testing system. Specifically, an on-board autonomous vehicle testing system is formed, which can be configured to perform one or more steps of one or more methods described below. In one embodiment, at least one of the one or more processors mounted on AV 16, which forms part of the on-board autonomous vehicle testing system, is a GPU. The one or more processors of the on-board autonomous vehicle testing system may include processor 40 of the on-board vehicle computer 30. However, in other embodiments, one or more other processors of the vehicle electronics 18 are used instead of or supplement to processor 40, for example, processors independent of the on-board vehicle computer 30.

[0026] Any one or more of the processors discussed herein can be implemented as any suitable electronic hardware capable of processing computer instructions, and can be selected based on the application to which they will be used. Examples of usable processor types include a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a microprocessor, a microcontroller, etc. Any one or more of the non-transitory computer-readable storage discussed herein can be implemented as any suitable type of memory capable of storing data or information in a non-volatile manner and in electronic form, such that the stored data or information can be consumed by the processor. This memory can be any number of different types of electronic storage, and can be selected based on the application to which it will be used. Examples of usable memory types include magnetic or optical disk drives, ROM (read-only memory), solid-state drives (SSDs) (including other solid-state storage such as hybrid solid-state drives (SSHDs), other types of flash memory, hard disk drives (HDDs), non-volatile random access memory (NVRAM), etc. It should be understood that any one or more computers discussed herein may include other storage, such as volatile RAM used by the processor and / or multiple processors.

[0027] refer to Figure 2 It illustrates the operating environment, which includes a communication system 10' (for test site 12), a test site server 14, an AV 16 with vehicle electronics 18, a terrestrial network 20, a wireless carrier system 22, a GNSS satellite constellation 24, an onboard stand-alone computer 31, and wires and cables 33 for connecting the onboard stand-alone computer 31 to the vehicle electronics 18. The communication system 10' and the communication system 10 ( Figure 1 Similar to the communication system 10', it also includes an onboard independent computer 31 and wires and cables 33.

[0028] As shown in Figure 19, the vehicle-mounted standalone computer 31 is an onboard computer integrated into the AV 16. Since the vehicle-mounted standalone computer 31 is not part of the vehicle electronics 18 manufactured by the OEM, it is considered "standalone." The vehicle-mounted standalone computer 31 can be a mobile computer (e.g., a laptop, smartphone, personal computer) configured to perform the methods described below (e.g., methods 200 and 300). A wire cable 33 is used for coupling communication between the vehicle-mounted standalone computer 31 and the vehicle electronics 18, for example, coupling to the vehicle-mounted computer 30. The wire cable 33 can be, for example, a Universal Serial Bus (USB) cable, an Ethernet cable, or a cable interfaced to an on-board diagnostic port (e.g., the on-board diagnostic II (OBD II) port of the vehicle electronics 18). In other embodiments, the vehicle-mounted standalone computer 31 can be wirelessly coupled to the vehicle electronics 18, for example via SRWC technology or via a wireless carrier system 22. In the above embodiments, the wire cable 33 may be omitted. In at least one embodiment, the in-vehicle standalone computer 31 includes one or more processors and memory accessible by the one or more processors. The memory may include computer instructions that, when executed by the one or more processors of the in-vehicle standalone computer 31, implement method 200. Figure 3 ) and / or method 300 ( Figure 4 One or more processors and memories of the onboard stand-alone computer 31 may form (or may form partially) an onboard autonomous vehicle test system. In some embodiments, a combination of the onboard stand-alone computer 31 and one or more components of the vehicle electronics 18 (e.g., the onboard vehicle computer 30) may be used to form an autonomous vehicle test system and / or perform method 200, method 300 and / or one or more steps thereof.

[0029] refer to Figure 3This demonstrates an embodiment of a method 200 for enhancing environmental scenarios, for use during autonomous vehicle testing. According to at least some embodiments, method 200 generates a synthetic weather object augmentation image, which can then be used by AV 16 as sensor input during autonomous driving. In some such embodiments, autonomous vehicle processing is performed by AV 16 to determine the autonomous driving operation to be performed based on sensor input including the synthetic weather object augmentation image. In the embodiments discussed below, the vehicle's environment is a vehicle testing environment where experiments or tests are performed using one or more real autonomous vehicles. According to at least some embodiments, these tests or experiments involve incorporating the synthetic weather object augmentation image as input into one or more operations of the vehicle to achieve an effect consistent with the synthetic weather object augmentation image actually captured by camera 34. In one embodiment, method 200 is performed by a remotely based autonomous vehicle testing system, which may include one or more components of test site server 14 (and / or other computers / electronic devices remotely connected to AV 16). In another embodiment, method 200 is performed by an on-board autonomous vehicle testing system, which may include an on-board vehicle computer 30, other computers / components of vehicle electronics 18, and / or an on-board stand-alone computer 31.

[0030] Method 200 begins at step 210, where an image representing scene information is captured by AV 16. The image is captured by a camera mounted on AV 16, which may be one of cameras 34. The image represents scene information taken from the vehicle's perspective; an example of such an image is given in 210A. The image may be represented as a pixel array, where each pixel of the pixel array specifies a color that can be specified using RGB or CMYK. In some embodiments, the image is a grayscale image, with pixels containing grayscale pixel information. The image is provided to and obtained by an autonomous vehicle testing system, which is a system for enhancing the image representing scene information such that the image includes one or more enhanced graphics. In one embodiment, the autonomous vehicle testing system is implemented by one or more processors and memories remotely and / or not carried on AV 16, such as a test site server 14 or processors and memories locally connected to the test site server 14, or processors and memories used as part of a roadside device. In another embodiment, the autonomous vehicle testing system is implemented by one or more hardware components of the vehicle electronics 18 of AV 16. Method 200 continues to step 220.

[0031] In step 220, virtual object graphics representing one or more virtual objects are generated, and then the virtual object graphics are rendered on the image to obtain an object-enhanced image. An object-enhanced image is an image captured by a camera that represents a graphically enhanced version of one or more objects. Figure 3 An example of an object-enhanced image is given at 220A. For example... Figure 3 As shown, object-enhanced image 220A includes graphics representing three objects 222A, 222B, and 222C, where each object is a bus. The three objects 222A, 222B, and 222C are all overlaid on image 210A to form object-enhanced image 220A. Various different techniques can be used to generate these graphics, such as those described below regarding method 300 (…). Figure 4 The techniques described in the text. Method 200 continues to step 230.

[0032] In steps 230 to 240, graphics representing weather conditions are generated and rendered to obtain a synthetic weather object-enhanced image 240A. According to at least some embodiments, weather-related graphics as part of image 240A are generated in two steps: in step 230, a global scene graphic is generated based on object-enhanced image 220A, wherein the global scene graphic is rendered together with virtual object graphics to form global scene image 230A; in step 240, a weather dynamic effect graphic is generated, wherein the weather dynamic effect graphic represents at least one specific weather dynamic effect corresponding to the weather conditions of the global scene image, and when the weather dynamic effect graphic is combined with global scene image 230A, a synthetic weather object-enhanced image 240A is generated. Figure 3 In the example depicted, the weather conditions for the global scene image are snowing.

[0033] In step 230, a global scene image is generated based on the object augmentation image 220A. For example... Figure 3 As shown, the global scene image 230A is an object-enhanced image 220A, but snow covers the outer surfaces of various objects, such as areas along a road, portions of the road, and vehicle hoods. Additionally, the global scene image 230A includes adjustments for illumination or color corresponding to observations during snowfall. Various techniques for generating and incorporating these elements, which are part of the global scene graph, will be discussed below. The global scene graph represents the changes in the global scene image 230A incorporating these elements, and in at least one embodiment, the global scene graph forms a global scene layer. Method 200 continues to step 240.

[0034] In step 240, a synthetic weather object enhancement image 240A is generated by combining a global scene image 230A and a weather dynamic effect graphic representing at least one specific weather dynamic effect. In at least some embodiments, the at least one specific weather dynamic effect corresponds to the weather conditions of the global scene graphic. Figure 3 As shown, one of the at least one specific weather dynamic effects is snowfall, where snowflakes are added to the global scene image 230A to make it appear as if it is currently snowing. Another at least one specific weather dynamic effect represents the accumulation or buildup of snow on a portion of the image corresponding to the windshield. Here, both of these specific weather dynamic effects correspond to the weather conditions used to generate the global scene image, which are snowfall conditions. However, in other embodiments, one or more specific weather dynamic effects that do not correspond to the specific weather dynamic effects used to generate the global scene image can be generated. In this example, adding the generated weather dynamic effect graphics to the global scene image 230A produces a generated synthetic weather object enhancement image 240A. Method 200 ends here.

[0035] refer to Figure 4 , Figure 4 An embodiment of method 300 for enhancing environmental scenes is shown for use by autonomous vehicles during testing. The following discussion of method 300 provides further details regarding specific steps, and it should be understood that the above discussion of method 200 applies here to method 300 when it does not contradict the following discussion of method 300. Although steps 310 to 360 are described as being performed in a specific order, it should be understood that steps 310 to 360 can be performed in any technically feasible order. For example, in one embodiment, step 340 may be performed before or in conjunction with step 330. In such an example, a dynamic weather effect graphic may be generated in step 340, and subsequently or simultaneously, a global scene graphic may be generated in step 330. These graphics may be combined with each other and with virtual object graphics (step 320) to generate a synthetic weather object enhancement image (step 350).

[0036] Method 300 begins at step 310, wherein an image representing scene information is obtained from the autonomous vehicle. The image can be transmitted from camera 34 to the autonomous vehicle test system, and this can be performed, for example, by transmitting the image from the camera 34 capturing the image to the onboard vehicle computer 30 via communication bus 38, and / or via wireless carrier system 22 and terrestrial network 20 to test site server 14. In another embodiment, this can be performed by transmitting the image from the onboard vehicle computer 30 to an onboard stand-alone computer 31 via wire cable 33. This step is similar to step 210 of method 200, and this discussion is incorporated herein. Method 300 continues to step 320.

[0037] In at least some embodiments, the method further includes the step of obtaining camera pose information, which represents the camera's pose during image capture by the camera. In at least some embodiments, the pose information specifies the camera's orientation and position relative to AV 16 or relative to the Earth (i.e., global position, which may be based on GNSS data). The pose of the camera capturing the image can be represented in various ways. For example, when the camera's orientation and position are fixed relative to AV 16, the position and orientation of AV 16 can be used with information specifying the position / orientation relationship or offset between the AV 16's position / orientation device (e.g., GNSS receiver 32) and the camera. This camera position / orientation offset information can be predetermined based on vehicle specifications and / or through experience or measurement processes. This predetermined camera position / orientation offset information can be stored at AV 16 or test site server 14. In another embodiment, for example, where the camera's field of view is movable relative to AV 16, the camera pose information can be represented or based on a combination of GNSS information (or other vehicle position / orientation information) and camera position / orientation information, which can be obtained by an accelerometer or other inertial sensor integrated into the camera. Such embodiments may also employ specific predetermined camera position / orientation offset information. At least according to some embodiments, the step of obtaining camera pose information may be performed after step 310 and before step 320.

[0038] In at least some embodiments, the method further includes the step of obtaining vehicle pose information representing the vehicle's position and orientation. The vehicle pose information may be obtained from GNSS data from GNSS receiver 32 and / or inertial sensor information, which may be obtained, for example, from one or more accelerometers mounted on AV 16, which are part of vehicle electronics 18. The vehicle pose information may be provided to test site server 14 via wireless carrier system 22 and / or terrestrial network 20. In embodiments where method 300 is performed by an on-board autonomous vehicle test system (e.g., including onboard vehicle computer 30, other parts of vehicle electronics 18, and / or onboard stand-alone computer 31), the vehicle pose information and / or camera pose information may be provided to the on-board autonomous vehicle test system, for example, by wired communication from vehicle electronics 18 to onboard stand-alone computer 31 using wired communication via wired cable 33.

[0039] In at least one embodiment, step 310, the step of obtaining camera pose information and / or obtaining vehicle pose information, includes transmitting images, camera pose information, and / or vehicle pose information from AV 16 to test site server 14 using 5G wireless communication. This can be performed by cellular chipset 46 and wireless carrier system 22. By using 5G communication, images, camera pose information, and / or vehicle pose information can be transmitted to test site server 14 with minimal or relatively low latency. Furthermore, in such embodiments, at least according to some embodiments, low-latency traffic / environment simulation and image enhancement can be achieved using a computer with high computing power (typically more readily available on a remote server than on a vehicle).

[0040] In step 320, virtual object graphics representing one or more virtual objects are generated. When the virtual object graphics are rendered on an image (or otherwise combined with an image), an object-enhanced image is produced. Step 220 of method 200 is similar to this step, and this discussion is incorporated herein. The presence, location, and orientation of the virtual objects to be rendered can be determined based on traffic simulation performed on an autonomous vehicle testing system. Traffic simulation is used to simulate virtual road objects and / or users, including cars, bicycles, other vehicles, pedestrians, traffic signals, and / or other traffic-related virtual objects. In at least one embodiment, the traffic simulation is at least partially based on one or more artificial intelligence (AI) technologies. Figure 5 The diagram illustrates an example of an AI-driven object rendering technology stack or processing flow. The traffic simulator 402 illustrated, which generates and executes traffic simulations, includes a Simulation of Urban Mobility (SUMO) system. TMModule 404 is used to model a multimodal transport system including road vehicles, public transport, and pedestrians. The traffic simulator 402 also includes a map building and ground fitting module 406 for fitting virtual objects created by the SUMO module 404 to appropriate locations within the test site 12. In at least some embodiments, the map building and ground fitting module 406 generates a 3D map, which can be constructed using LiDAR SLAM technology (or LeGO-LOAM) to generate an environment-dense point cloud. Given these point clouds, the ground plane is fitted using ground point extraction techniques (or cloth-based filtering algorithms), according to at least some embodiments. The pose of the object to be rendered is estimated using the SUMO module 404 and the map building and ground fitting module 406, as shown at 408. In at least one embodiment, the SUMO module 404 provides the position and yaw angle of the object, and the map building and ground fitting module 406 provides pitch and roll angle information for a specified virtual object. However, in other embodiments, the SUMO module 404 can provide both pitch and roll angle information.

[0041] Image 410 received from AV 16, along with camera and vehicle pose information 412, combined with object pose estimation information 408, allows for the determination of a suitable appearance for rendering a virtual object on the image in step 414. Rendering the virtual object graphic onto the image yields an object-enhanced image 416. Known techniques can be used to determine the appearance of the virtual object, and once determined, a virtual object graphic representing the virtual object can be rendered on the image. Rendering of the virtual object can be performed using known techniques. In one embodiment, rendering of the object's appearance is performed using two deep learning-based algorithms, such as Visual Object Network (VON) and HoloGAN. In such an embodiment, the input to the neural network rendering includes latent factors controlling the shape and appearance of the virtual object, as well as camera / object pose information. The object-enhanced image can be further processed using an alpha blend of the original or initial image (e.g., represented by image data), and the graphic representing the virtual object can then be rendered over this processed background image. According to at least some embodiments, by using such techniques, virtual objects exhibit realism, as if they were real objects as part of the original image captured by the camera. Once the graph representing one or more objects has been rendered, method 300 continues to step 330.

[0042] In step 330, a global scene graph is generated, and in at least some embodiments, the global scene graph is generated based on an object-enhanced image. When the global scene graph is rendered on the image and / or rendered together with the object-enhanced image, a visual representation of the environment is produced that behaves the same way the environment does under predetermined or selected weather conditions. In at least some embodiments, the predetermined weather conditions may be selected by a user (e.g., a test site operator or other personnel) or by a simulator or other automated computer process. In at least one embodiment, the weather conditions may be selected based on the test configuration or scenario. Furthermore, in at least some embodiments, the pre-selected weather conditions are different from, or at least may be different from, the current weather conditions of the environment in which the AV 16 is being tested.

[0043] In at least one embodiment, a global scene graph is generated using artificial intelligence (AI) technology. AI technology allows for global modifications to the image, such as changes in illumination, the addition of moisture or precipitation on roads (e.g., wet road appearance, snow covering roads), etc. In one embodiment, the AI ​​technology refers to multimodal unsupervised image-to-image translation (MUNIT) technology, which employs the MUNIT algorithm and is used to render the global scene graph. In at least some embodiments, the MUNIT implementation assumes that the latent space of the image is decomposed into a content space and a style space. Images from different domains share a common content space reflecting the underlying spatial structure, but each has a unique style space containing scene variations, such as weather conditions. MUNIT is trained in an unsupervised manner, requiring only unpaired images under different weather conditions. According to some embodiments, three models (day to night, sunny to wet weather, and sunny to snowy weather) are trained on collected data, which may include data collected from a test site or environment. Figure 6 As shown, input image 502 was taken on a sunny day, and the image was modified using the first AI model (as shown in 504) to make it appear as if it were taken at night with lower illumination (see image 520). Furthermore, as... Figure 6 As shown, the image is modified using a second AI model to make the road appear wet, as shown at position 506 (see image 522). Additionally, as... Figure 6As shown, the image is modified using a third AI model to make it appear as if the road is covered in snow, as shown at 508 (see image 532). Although specific exemplary weather and lighting conditions are described herein, it should be understood that other weather and lighting conditions, such as foggy weather or nighttime lighting, can be used. Furthermore, in some embodiments, in addition to or instead of rendering weather-related graphics as part of the global scene graphics, lighting graphics or effects can be generated and / or rendered as part of the global scene image. Thus, in some embodiments, the global scene graphics include graphics that incorporate predetermined lighting effects into the composite weather object enhancement image. Method 300 continues to step 340.

[0044] In step 340, a weather dynamic effect graphic is generated. The weather dynamic effect graphic represents at least one specific weather dynamic effect, and in at least some embodiments, the at least one specific weather dynamic effect corresponds to predetermined weather conditions in the global scene image. Furthermore, in at least some embodiments, the weather dynamic effect graphic is generated using physically based computational techniques. Figure 6 As shown, physics-based models or techniques are used to generate weather dynamic effects graphics representing rain lines 510 and snowflakes 512, or at least those associated with them. Specifically, specific weather dynamic effects associated with rain lines 510 include the addition of raindrops 514, and may also include the effect of blurred raindrops / rain lines 516 and / or blurred environment 518. In some embodiments, AI-based techniques may be used instead of or to assist physics-based techniques in generating weather dynamic effects graphics. Various examples of such specific weather dynamic effects are shown at 524, 526, 528, and 530.

[0045] In one embodiment, a particle system (e.g., from Unity) can be used to simulate the spatiotemporal distribution and trajectory of raindrops in three-dimensional space. TM The rain line rendering is performed using the API. The rain line texture is rendered by or based on a line database, and the appearance of the rain lines is generated based on specific camera / image parameters, such as light sources and raindrop size. The final rendered image shown at 524 is the global scene image with the rendered rain line layer on top of it.

[0046] In one embodiment, raindrop rendering can be performed based on one of two scenarios: in the first scenario, the camera is focused on the environment; in the second scenario, the camera is focused on the raindrops. In the first scenario, the raindrops are blurred, which can be simulated by generating raindrops using a spherical raindrop model and then modifying the appearance of the raindrops using a fisheye effect. An example of this is shown at 526. In the second scenario, a sharp image of the raindrops taken from the real world is obtained and blended with a defocused, blurred version of the image using an alpha blending algorithm, an example of which is shown at 528.

[0047] In one embodiment, using Snow100k 2 Realistic snowflake images or graphics from a dataset are used to perform snowflake rendering. According to one embodiment, snowflakes are rendered from images or graphics contained in Snow100k. 2 Masked snowflake images are extracted from real images in the dataset, and a set of parameters (e.g., size, speed) are assigned to each image based on the density of the snowflakes to be simulated. For a real image J from a vehicle, a snowflake layer S that meets the rendering requirements is selected, and then cropped / resized to fit the input frame. The final rendered result I is synthesized using the snowflake image model: I = zS + J(1-z), where z is the binary mask of S. An example of this result is shown at 530. Method 300 continues to step 350.

[0048] In step 350, a composite weather object enhancement image is generated based on virtual object graphics, global scene graphics, and dynamic weather effect graphics. In one embodiment, the object enhancement image is first generated by rendering virtual object graphics on an image obtained from a camera (or alternatively combining virtual object graphics with an image obtained from a camera), for example in 220A (…). Figure 3 The image shown at ( ). Then, the global scene graph is added to or combined with the object enhancement image (or virtual object graph) to generate a global scene image, such as global scene image 230A ( ). Figure 3 Subsequently, dynamic weather effects graphics are added to or combined with the global scene image to generate composite weather object enhancement images, such as in 240A (…). Figure 3 The image shown is located at ( ). In one embodiment, a weather motion graphics is generated to form a weather motion graphics layer, which is then rendered on a global scene image to produce a synthetic weather object-enhanced image. In other embodiments, the global scene graphics and the weather motion graphics may first be combined with each other to form a weather layer, which may then be combined with the object-enhanced image to generate a synthetic weather object-enhanced image. Of course, according to other embodiments, different order of steps may be used to generate the synthetic weather object-enhanced image. Furthermore, in one embodiment, the synthetic weather object-enhanced image includes only the global scene graphics or the weather motion graphics, but not both. Furthermore, in another embodiment, a weather-enhanced image with the global scene graphics and / or the weather motion graphics is generated instead of generating a synthetic weather object-enhanced image. The weather-enhanced image may not include any virtual objects, in which case step 320 may be omitted. Method 300 continues to step 360.

[0049] In step 360, a synthetic weather object augmentation image (or other augmentation image) is input to the onboard vehicle controller of the autonomous vehicle. In at least some embodiments, the synthetic weather object augmentation image is input to the onboard vehicle controller, causing the AV to perform at least one autonomous driving operation based on the synthetic weather object augmentation image. Furthermore, in at least one embodiment, the synthetic weather object augmentation image is input to the onboard vehicle controller in a manner that makes it appear as if the synthetic weather object augmentation image were a non-augmented image. If the synthetic weather object augmentation image is generated at a remote autonomous vehicle testing system, the synthetic weather object augmentation image can be transmitted to the AV 16 from a remote system (e.g., test site server 14) via 5G communication, which can be performed by the wireless carrier system 22 and the cellular chipset 46. Of course, in other embodiments, other communication paths can be used to transmit the synthetic weather object augmentation image to the AV 16, such as via the terrestrial network 20, roadside equipment, and SRWC line 44.

[0050] As described above, the synthetic weather object augmentation image is input to the vehicle controller in such a way that the vehicle controller treats it as if it were a non-augmented image. A non-augmented image refers to an image captured by a camera and not modified to include any virtual graphics or weather / environmental changes, such as the addition of snow or changes in illumination. AV16 can be configured to pass the synthetic weather object augmentation image to the vehicle controller (e.g., vehicle computer 30) in such a way that the vehicle controller treats it as if it were an image containing only real elements. This allows for scene augmentation during testing by rendering various weather and traffic scenarios, regardless of the actual weather conditions at the test site 12.

[0051] Furthermore, in some embodiments, the vehicle electronics 18 can be modified to make the AV 16 behave as if the virtual object actually exists and / or make the predetermined weather conditions actually exist at the test site 12. For example, when a virtual vehicle 222A is added to the synthetic weather object enhancement image 240A, the vehicle electronics 18 can modify the lidar signal so that the lidar signal indicates the virtual vehicle 222A actually exists according to the synthetic weather object enhancement image 240A. Additionally, in one embodiment, the modified or virtual lidar signal is received by the test site server 14 and input to the vehicle controller such that the vehicle controller behaves as if the modified lidar signal was actually generated from the lidar sensor 36 (rather than being modified). This concludes method 300. In at least some embodiments, continuous execution of method 300 can generate a series of synthetic weather object enhancement images and input them to the vehicle controller.

[0052] It should be understood that the foregoing description is one or more embodiments of the present invention. The present invention is not limited to the specific embodiments disclosed herein, but is defined only by the following claims. Furthermore, the statements contained in the foregoing description relate to the disclosed embodiments and should not be construed as limiting the scope of the invention or the terminology used in the claims, unless the terms or phrases are expressly defined above. Various other embodiments, as well as various changes and modifications to the disclosed embodiments, will become apparent to those skilled in the art.

[0053] As used in this specification and claims, the terms “such as,” “e.g.,” “for example,” “like,” “as,” and “etc.,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, when used in conjunction with a list of one or more components or other items, shall each be interpreted as open-ended, meaning that the list should not be considered to exclude other additional components or items. Other terms shall be interpreted using their broadest reasonable meaning unless they are used in a context requiring a different interpretation. Furthermore, the term “and / or” shall be interpreted as encompassing or. Thus, for example, the phrase “A, B, and / or C” shall be interpreted to cover all of the following: “A”; “B”; “C”; “A and B”; “A and C”; “B and C”; and “A, B, and C”.

Claims

1. A method for enhancing environmental scenarios for use by autonomous vehicles during testing, the method comprising the following steps: Images representing scene information are obtained from an autonomous vehicle, wherein the images are captured by cameras mounted on the autonomous vehicle and depict the environment in which the autonomous vehicle is driving; Generate virtual object graphics representing one or more virtual objects, and when the virtual object graphics are rendered on the image, produce an object-enhanced image; Based on the enhanced image of the object, a global scene graphic representing the macroscopic static effect of the weather is generated to simulate the macroscopic visual effect of the environment under predetermined weather conditions. The global scene graphic is generated by one or more artificial intelligence (AI) technologies. Generate a weather dynamic effect graphic representing at least one specific weather dynamic effect, wherein the weather dynamic effect graphic is generated by one or more computational techniques based on a physical model; Based on the virtual object graphics, the global scene graphics, and the dynamic weather effect graphics, a synthetic weather object enhanced image is generated; and The synthetic weather object enhancement image is input into the vehicle controller of the autonomous vehicle, so that the autonomous vehicle performs at least one autonomous driving operation based on the synthetic weather object enhancement image; The method further includes the step of obtaining camera pose information, the camera pose information representing the pose of the camera during the period when the image is captured by the camera, and wherein the pose of the one or more objects is determined based on the camera pose information such that the one or more objects appear realistic when rendered on the image; The method further includes the step of obtaining vehicle pose information, which represents the pose of the autonomous vehicle during the period when the image is captured by the camera; The camera pose information and / or the vehicle pose information are used to generate the virtual object graphics; The generation of virtual object graphics representing one or more virtual objects includes: rendering the appearance of the objects using a deep learning-based algorithm; wherein the input to the neural network rendering includes latent factors that control the shape and appearance of the virtual objects and camera / object pose information. The generation of global scene graphics representing the macroscopic static effects of weather includes: modifying the image using a first AI model to make it appear as if it were a nighttime event with low illumination; modifying the image using a second AI model to make the road appear as if it were wet; and modifying the image using a third AI model to make the road appear as if it were covered in snow.

2. The method according to claim 1, wherein, The virtual object graphic is generated based on at least one AI technology.

3. The method according to claim 1, wherein, The method is executed by an onboard independent computer, wherein the onboard independent computer is connected to and communicates with the vehicle electronics of the autonomous vehicle.

4. The method according to claim 1, wherein, Each step of the method is performed more than once to generate multiple synthetic weather object enhancement images and input the multiple synthetic weather object enhancement images into the on-board vehicle controller.

5. The method according to claim 1, wherein, The images are acquired at the autonomous vehicle and transmitted from the autonomous vehicle to a test site server, wherein the transmission of the images includes the use of 5G cellular communication.

6. The method according to claim 5, wherein, The test site server is configured to perform the following steps: generating virtual object graphics, generating global scene graphics, generating dynamic weather effect graphics, generating synthetic weather object enhancement images, and inputting the synthetic weather object enhancement images into the on-board vehicle controller of the autonomous vehicle.

7. The method according to claim 6, wherein, The step of inputting the synthetic weather object enhancement image into the on-board vehicle controller of the autonomous vehicle includes: sending the synthetic weather object enhancement image to the autonomous vehicle using 5G cellular communication.

8. The method according to claim 1, wherein, The at least one specific weather dynamic effect corresponds to the predetermined weather conditions.

9. The method according to claim 1, wherein, The synthetic weather object enhancement image is input into the vehicle controller in such a way that the vehicle controller behaves as if the synthetic weather object enhancement image were a non-enhanced image.

10. The method of claim 1, wherein the predetermined weather conditions are different from the current weather conditions of the environment, such that when the global scene graphics are rendered together with the object enhancement image, the visual representation of the environment is as it would behave if the environment were experiencing weather different from the current weather conditions of the environment.

11. An autonomous vehicle testing system, comprising one or more electronic processors and a non-transitory computer-readable memory, the non-transitory computer-readable memory being accessible by the one or more electronic processors and storing computer instructions; in, When the computer instructions are executed by the one or more electronic processors, the autonomous vehicle testing system: Images representing scene information are obtained from an autonomous vehicle, wherein the images are captured by cameras mounted on the autonomous vehicle and depict the environment in which the autonomous vehicle is driving; Generate virtual object graphics representing one or more virtual objects, and when the virtual object graphics are rendered on the image, produce an object-enhanced image; Based on the enhanced image of the object, a global scene graphic representing the macroscopic static effect of the weather is generated to simulate the macroscopic visual effect of the environment under predetermined weather conditions. The global scene graphic is generated by one or more artificial intelligence (AI) technologies. Generate a weather dynamic effect graphic representing at least one specific weather dynamic effect, wherein the weather dynamic effect graphic is generated by one or more computational techniques based on a physical model; Based on the virtual object graphics, the global scene graphics, and the dynamic weather effect graphics, a synthetic weather object enhanced image is generated; and The synthetic weather object enhancement image is input into the vehicle controller of the autonomous vehicle, so that the autonomous vehicle performs at least one autonomous driving operation based on the synthetic weather object enhancement image; The autonomous vehicle testing system is further configured such that, when the one or more electronic processors execute the computer instructions, the autonomous vehicle testing system can also obtain camera pose information, the camera pose information representing the pose of the camera during the period when the image is captured by the camera, and wherein the pose of the one or more objects is determined based on the camera pose information, so that the one or more objects appear realistic when rendered on the image. The autonomous vehicle testing system is further configured such that, when the computer instructions are executed by the one or more electronic processors, the autonomous vehicle testing system also obtains vehicle pose information, the vehicle pose information representing the pose of the autonomous vehicle during the period when the image is captured by the camera; The camera pose information and / or the vehicle pose information are used to generate the virtual object graphics; The generation of virtual object graphics representing one or more virtual objects includes: rendering the appearance of the objects through a Visual Object Network (VON) and HoloGAN; wherein the input to the neural network rendering includes latent factors that control the shape and appearance of the virtual objects and camera / object pose information. The generation of global scene graphics representing the macroscopic static effects of weather includes: modifying the image using a first AI model to make it appear as if it were a nighttime event with low illumination; modifying the image using a second AI model to make the road appear as if it were wet; and modifying the image using a third AI model to make the road appear as if it were covered in snow.

12. The autonomous vehicle testing system according to claim 11, wherein, At least one of the one or more electronic processors is a graphics processing unit (GPU).

13. The autonomous vehicle testing system according to claim 11, wherein, The one or more electronic processors are installed at a location remote from the autonomous vehicle, and the autonomous vehicle is configured to use 5G cellular communication to send the images to a test site server and receive the synthetic weather object augmentation images from the test site server.

14. The autonomous vehicle testing system according to claim 11, wherein, The global scene graphics include graphics in which predetermined lighting effects are added to the synthetic weather object enhancement image.

Images