Method and system for stitching images into a virtual image

Abstract

A system and method for assisting an operator in operating a vehicle is provided. In one embodiment, the system includes a sensor system configured to generate sensor data sensed from a vehicle environment. The system also includes a control module configured to determine a scene of the environment based on the sensor data via a processor, store a shape of at least one feature in the scene, modify video data based on the stored shape, and present the modified video data to an operator for display.

Description

Technical Field

[0001] The technical field generally relates to operator assistance systems for vehicles and, more particularly, to methods and systems for providing a virtual image of a vehicle environment. Background Art

[0002] A vehicle may incorporate and use a variety of auxiliary devices to assist the operator. For example, various sensors may be arranged at various locations on the vehicle. The various sensors sense observable conditions of the vehicle's environment. For example, multiple cameras or other sensors may sense the conditions of the road or environment on which the vehicle is traveling or about to travel. In some instances, it is desirable to present an under-the-hood or virtual view of the image to the operator. In this case, the images provided by the various cameras must be stitched together to create a virtual view.

[0003] Therefore, it is desirable to provide methods and systems for improved stitching of multiple images into a single virtual image.Other desirable features and characteristics of the embodiments described herein will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background. Summary of the invention

[0004] In one exemplary embodiment, a system and method for assisting an operator in operating a vehicle is provided. In one embodiment, the system includes a sensor system configured to generate sensor data sensed from a vehicle environment. The system also includes a control module configured to determine a scene of the environment based on the sensor data via a processor, store a shape of at least one feature in the scene, modify video data based on the stored shape, and present the modified video data to the operator for display.

[0005] In various embodiments, the sensor system includes one or more cameras.

[0006] In various embodiments, at least one feature includes printed markings on the ground.

[0007] In various embodiments, the control module is further configured to, via the processor, stitch the sensor data to form a scene of the environment and identify errors in features of the stitched sensor data, wherein the video data is modified based on the identified errors. In various embodiments, the control module is further configured to match the features with errors to stored shapes.

[0008] In various embodiments, the control module is configured, via the processor, to modify the video data by warping a ground plane of one or more portions of the spliced ​​scene data based on a stored shape.

[0009] In various embodiments, the control module is configured to modify the video data by smoothing the lines in the distorted ground plane. In various embodiments, the control module is further configured to modify the video data by sharpening the edges of the smoothed lines in the distorted ground plane. In various embodiments, the control module is further configured to splice the sensor data and the stored shapes to modify the video data.

[0010] In various embodiments, the control module is configured to calculate a confidence score based on at least one of the vehicle speed data, the steering angle data, and the real geometry data, and to modify the video data by stitching the video data based on the stored shape and the confidence score.

[0011] In another embodiment, a method includes: receiving sensor data from a sensor system that senses an environment of a vehicle; determining, by a processor, a scene of the environment based on the sensor data; determining, by the processor, a shape of a feature in the environment based on the sensor data; modifying, by the processor, video data based on the shape of the feature in the environment; and generating display data to display the modified video data for viewing by an operator of the vehicle.

[0012] In various embodiments, the sensor system includes one or more cameras.

[0013] In various embodiments, the at least one feature comprises a marking on a ground plane.

[0014] In various embodiments, the method further includes stitching the sensor data to form a scene of the environment, and identifying errors in features of the stitched sensor data, wherein the video data is modified based on the identified errors. In various embodiments, the method further includes matching the features with errors to stored shapes.

[0015] In various embodiments, the method further comprises modifying the video data by warping a ground plane of one or more portions of the spliced ​​scene data based on a stored shape. In various embodiments, the method further comprises modifying the video data by smoothing lines in the warped ground plane. In various embodiments, the method further comprises modifying the video data by sharpening edges of smoothed lines in the warped ground plane. In various embodiments, the method further comprises splicing the sensor data and the stored shape to modify the video data.

[0016] In various embodiments, the method further includes calculating a confidence score based on at least one of the vehicle speed data, the steering angle data, and the real-world geometry data, wherein the method modifies the video data by stitching the video data based on the stored shapes and the confidence scores. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Exemplary embodiments will be described below with reference to the following drawings, wherein like numerals represent like elements, and wherein:

[0018] Figure 1 is a top view schematic diagram of a vehicle with a virtual reality system according to various embodiments;

[0019] Figure 2 is a functional block diagram illustrating a virtual reality system according to various embodiments;

[0020] Figure 3 is an illustration of a display of a virtual reality system according to various embodiments;

[0021] Figure 4 is a data flow diagram illustrating a control module of a virtual reality system according to various embodiments; and

[0022] Figure 5 and 6 is a flow chart illustrating a method of controlling content to be displayed on a display screen of a virtual reality system according to various embodiments. DETAILED DESCRIPTION

[0023] The following detailed description is for illustrative purposes only and is not intended to limit application and use. In addition, it is not subject to any express or implied theory presented in the above technical field, background, summary or the following detailed description. It should be understood that in all the figures, corresponding reference numerals represent similar or corresponding parts and features. As used herein, the term system or module may refer to any combination or collection of mechanical and electrical hardware, software, firmware, electronic control components, processing logic and / or processor devices, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups), memories containing one or more executable software or firmware programs and related data, combinational logic circuits and / or other suitable components that provide the functions.

[0024] Embodiments may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by any number, combination, or collection of mechanical and electrical hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of the present invention may employ various combinations of mechanical components, such as traction devices, indicators, or signaling devices; and electronic components, such as integrated circuit components, memory elements, digital signal processing elements, logic elements, lookup tables, imaging systems and devices, etc., which may perform various functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that the embodiments described herein may be implemented in conjunction with any number of mechanical and / or electronic systems, and that the vehicle systems described herein are merely exemplary.

[0025] For the sake of brevity, conventional components and techniques of the system and other functional aspects (as well as the individual operating components of the system) are not described in detail herein. In addition, the connecting lines shown in the various figures contained herein are intended to represent example functional relationships and / or physical connections between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in embodiments of the present invention.

[0026] Figure 1 1 is an illustration of a top view of a vehicle, shown generally at 10, equipped with a virtual reality system, shown generally at 12, according to various embodiments. As will be discussed in greater detail below, the virtual reality system 12 generally uses data from a sensor system 14 of the vehicle 10 and customizable software to allow a user to experience a virtual reality of features beneath the vehicle 10. As used herein, the term "virtual reality" refers to a replication of a real or imagined environment and / or component. For example, the virtual reality system 12 may be implemented to provide a visualization of features beneath the vehicle 10. In such an example, the display screen 16 ( Figure 2 ) can be placed anywhere on the vehicle 10 and can, for example, display images and / or videos that create a virtual reality of the bottom of the vehicle 10, as the vehicle is not visible under the hood or trunk of the vehicle.

[0027] Although the discussion herein is with respect to vehicle 10 being a passenger vehicle, it should be understood that the teachings herein are compatible with all types of automobiles, including but not limited to sedans, coupes, sport utility vehicles, pickup trucks, minivans, full-size vans, trucks, buses, and any type of tractor-trailer, including trailers.

[0028] like Figure 1 As shown, vehicle 10 generally includes a body 13, front wheels 18, rear wheels 20, a suspension system 21, a steering system 22, and a propulsion system 24. Wheels 18-20 are rotationally coupled to vehicle 10 near respective corners of body 13. Wheels 18-20 are coupled to body 13 via suspension system 21. Wheels 18 and / or 20 are driven by propulsion system 24. Wheel 18 can be steered by steering system 22.

[0029] The body 13 is disposed on or incorporated with a chassis (not shown) and substantially encloses the components of the vehicle 10. The body 13 is configured to separate a powertrain compartment 28 (including at least a propulsion system 24) from a passenger compartment 30, which includes seating (not shown) for one or more passengers of the vehicle 10. As used herein, components "underneath" the vehicle 10 are components disposed below the body 13, such as, but not limited to, wheels 18 and 20 (including their respective tires) and suspension system 21.

[0030] The vehicle 10 also includes a sensor system 14 and an operator selection device 15. The sensor system 14 includes one or more sensing devices that sense observable conditions of components of the vehicle 10 and / or sense observable conditions of the external environment of the vehicle 10. The sensing devices may include, but are not limited to, radars, lidars, global positioning systems, optical cameras, thermal cameras, ultrasonic sensors, height sensors, pressure sensors, steering angle sensors, and / or other sensors. The operator selection device 15 includes one or more user-manipulable devices that the user can manipulate to provide input. For example, the input may involve activation of the display of virtual reality content and the desired viewing angle of the content to be displayed. The operator selection device 15 may include knobs, switches, touch screens, voice recognition modules, etc.

[0031] like Figure 2 As shown in more detail in Figure 1 , the virtual reality system 12 includes a display screen 32 that is communicatively coupled to a control module 34. The control module 34 is communicatively coupled to the sensor system 14 and the operator selection device 15.

[0032] Display screen 32 may be disposed at a location within passenger compartment 30 that is viewable by an operator of vehicle 10. For example, display screen 32 may be incorporated with an infotainment system (not shown) or an instrument panel (not shown) of vehicle 10. Display screen 32 displays content that allows a viewer to experience virtual reality. For example, Figure 3 As shown, in various embodiments, the content 42 includes graphics of vehicle components, graphics of terrain features, and a depiction of a scene 48 in which the vehicle 10 is traveling, including the ground, curbs, road markings, buildings, and the like.

[0033] Virtual reality content 42 may be displayed in real time and / or predefined. Figure 3 As shown, a scene of an environment is generated by stitching together sensor data from one or more sensors. Thereafter, a virtual image of the front tire is imposed on the scene of the environment to create a virtual image that depicts what is under the hood and reveals the terrain. The stitched scene and / or virtual image is presented in an improved manner based on features extracted from the live scene. For example, as will be discussed in more detail below, the shapes of the features are extracted from the live scene and stored. When the scene is stitched, the stitching is improved by using the stored features. The final stitched image is then presented to the operator in a manner that maintains the stored shapes of the features.

[0034] Back to Figure 1, the control module 34 may be dedicated to the display screen 32, may control the display screen 32 and other features of the vehicle 10 (e.g., a body control module, an instrument control module, or other feature control module), and / or may be implemented as a combination of control modules that control the display screen 32 and other features of the vehicle 10. For exemplary purposes, the control module 34 will be discussed and illustrated as a single control module dedicated to the display screen 32. The control module 34 directly controls the display screen 32 and / or transmits data to the display screen 32 so that virtual reality content can be displayed.

[0035] The control module 34 includes at least a memory 36 and a processor 38. As will be discussed in more detail below, the control module 34 includes instructions that, when processed by the processor 38, control what is to be displayed on the display screen 32 based on sensor data received from the sensor system 14 and user input received from the operator selection device 15. The control module also includes instructions that, when processed by the processor 38, control what is to be displayed based on the stored shape of the video and features, which will be described in more detail below.

[0036] Reference now Figure 4 And continue to refer to Figure 1-3 , the data flow diagram shows various embodiments of the control module 34 in more detail. Various embodiments of the control module 34 according to the present invention may include any number of submodules. As can be appreciated, Figure 4 The submodules shown may be combined and / or further divided to similarly generate virtual reality content for the operator to view. The inputs to the control module 34 may be received from the sensor system 14, from the operator selection device 15, from other control modules (not shown) of the vehicle 10, and / or determined by other submodules (not shown) of the control module 34.

[0037] In various embodiments, the control module 34 includes a shape determination module 50 , a splicing module 52 , a vehicle component determination module 54 , a display determination module 56 , a graphics data memory 58 , and a feature data memory 60 .

[0038] The graphic data memory 58 receives and stores graphics of various features of the vehicle 10, such as features underneath the vehicle 10, including the front tires 18, the rear tires 20, suspension system components, etc. Figure 3 The feature data storage 60 receives and stores shape data of various features of the scene in the environment of the vehicle 10 determined by the shape determination module 50 .

[0039] The shape determination module 50 receives sensor data 62 as input. The shape determination module 50 extracts a frame of a scene depicting an environment from the sensor data 62. The shape determination module 50 identifies key features, such as key points or lines of a shape (e.g., markings on a road, such as lines, multiple lines, dashed lines, intersections of lines, straight lines, curves, circles, etc.) from the scene in the frame. The shape determination module 50 calculates a confidence score for the identified shape data. For example, the shape determination module 50 calculates one or more confidence scores based on the current steering wheel data 64, the vehicle speed data 66, and the real-world geometry data 68. The shape determination module 50 then uses the confidence scores to calculate weights associated with models (e.g., geometric models, feature fitting models, etc.) used in image-based stitching. For example, as the parameters (e.g., steering wheel data 64, vehicle speed data 66, and real-world geometry data 68) increase, the weight associated with one of the models (e.g., geometric model) decreases, while the weight associated with another model (e.g., feature fitting model) increases. The shape determination module 50 then calculates the coordinates of the shape relative to the vehicle (eg, in a vehicle coordinate system) based on the confidence scores and weights, and stores the shape data in the feature data memory 60 .

[0040] The stitching module 52 receives sensor data 62 as input. The stitching module 52 generates a scene of the environment based on the sensor data 62. For example, the sensor data 62 may include image data or video data provided by multiple cameras arranged around the vehicle 10, and the stitching module 52 stitches the data 62 to generate a scene (e.g., a 360-degree view or other view of the environment). The stitching module 52 stitches the scene with stored images of stored vehicle components.

[0041] Thereafter, the stitching module 52 processes the stitched scene to determine the content. When stitching errors are found (eg, broken or skewed lines), the stitching module 52 adjusts the stitched image of the scene based on the shape data 70 stored in the feature data memory 60 .

[0042] For example, the stitching module 52 processes the scene image to remove any shadows in the scene, and then matches features from the stitched image of the scene with features identified by the shape data 70 stored in the feature data memory 60. The stitching module 52 uses the weights and one or more models as described above to adjust the stitching between the stored data and the real-time image of the scene to maintain the matching shape data 70. For example, the stitching module 52 adjusts portions of the stored data and applies shifts to different stored portions in the stitched image to maintain the shape and coordinates of the shape data 70, thereby producing a blended transparent area; a filter (such as, but not limited to, a harmonic average filter) is then used to smooth the junction area between the real-time data and the stored data. Optionally, the stitching module sharpens the inner edges of any contours. The stitching module then generates the final stitched scene as scene data 72.

[0043] The vehicle component determination module 54 receives as input the sensor data 74. The vehicle component determination module 54 processes the sensor data 74 to determine the actual positions of various vehicle features. For example, the sensor data 74 may include height data, pressure data, etc. from the vehicle body 13 and / or the suspension system 21. The vehicle component determination module 54 generates vehicle data 76 based on the actual positions of the vehicle features.

[0044] The display determination module 56 receives as input the scene data 72, the vehicle data 76, and the user input data 78. Based on the received data, the display determination module 56 generates display data 80 to display the content 42 including the enhanced scene and vehicle features as a virtual reality to the operator.

[0045] For example, when the vehicle 10 is traveling, the display determination module 56 embeds a graphic 82 retrieved from the graphic data storage 58 that shows under-vehicle features in the scene based on the current position. The display determination module 56 retrieves the graphic 82 of the vehicle features from the graphic data storage 58 based on the selected perspective indicated by the user input data 78. The display determination module 56 then overlays the changed vehicle feature graphics on the scene indicated by the scene data 72. The display determination module 56 then generates display data 80 including the scene and vehicle component graphics.

[0046] Reference now Figure 5 and 6 , and continue to refer to Figure 1-4 , the flowchart illustrates methods 100, 200 that may be performed by the virtual reality system 12 according to various embodiments. It is understood that the order of operations within the methods 100, 200 is not limited to the following. Figure 5 and 6 The present invention does not intend to perform the above steps in the order shown, but may be performed in one or more different orders according to the present invention.

[0047] As further understood, Figure 5 and 6 The methods 100 , 200 may be scheduled to run at predetermined time intervals during operation of the vehicle 10 , and / or may be scheduled to run based on predetermined events.

[0048] like Figure 5 As shown, the method 100 determines and stores shape data for correcting the stitching scene. In various embodiments, the method 100 may be performed by Figure 1 and 2 The control module 34 is executed, more specifically, by Figure 4The shape determination module 50 is executed. In one example, the method may start at 105. Camera input data is received at 110, and a frame is extracted at 120. Features of the frame are evaluated at 130, such as lines or other key points. The coordinates of the features are converted to the vehicle coordinate system at 140. Confidence scores and weights of the feature coordinates are calculated based on the model prediction at 150. A final transformation of the coordinates is calculated at 160 based on the confidence scores. Thereafter, the final coordinates are stored as shape data in a feature data memory at 170; and the method may end at 180.

[0049] like Figure 6 As shown, method 200 uses shape data to correct errors in stitching. In various embodiments, method 200 may be performed by Figure 1 and 2 The control module 34 is executed, more specifically, by Figure 4 The stitching module 52 is executed. In one example, the method may start from 205. At 210, camera input data is received, and at 220, frames are extracted.

[0050] Top-down projection and stitching of frames is performed at 230. The projected frames are checked for errors at 240. When no errors are found at 240, method 200 continues at 220 to obtain the next frame from sensor data.

[0051] When an error is found at 240, a shape associated with the error is matched with a shape stored in a feature data memory at 250. At 260, the frame is coordinate transformed based on the vehicle odometer. At 270, any shadows present in the frame are removed. At 280, the coordinates of the frame corresponding to the error are transformed based on the matched shape data. Any lines associated with the shape are smoothed at 290, and the inner edges of any contours are sharpened at 300. Thereafter, at 310, the final improved scene data is provided for display by a display device; and the method can end at 320.

[0052] Although at least one exemplary embodiment has been given in the foregoing detailed description, it should be understood that there are a large number of variations. It should also be understood that one or more exemplary embodiments are only examples and are not intended to limit the scope, applicability or configuration of the present invention in any way. On the contrary, the above detailed description will provide a convenient roadmap for implementing one or more exemplary embodiments to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the scope of the disclosure described in the attached claims and their legal equivalents.

Claims

1. A system for assisting an operator in operating a vehicle, comprising: a sensor system configured to generate sensor data sensed from an environment of the vehicle; and A control module is configured to determine, via a processor, a scene of an environment based on the sensor data, identify coordinates of at least one feature within the scene, calculate a confidence score for the coordinates based on at least one of vehicle speed data, steering angle data, and real-world geometry data, store the coordinates as feature coordinates of at least one feature in the scene based on the confidence score, modify video data by distorting a ground plane of one or more portions of the spliced ​​scene data based on the stored feature coordinates, and present the modified video data to an operator for display.

2. The system according to claim 1, wherein: The sensor system includes one or more cameras.

3. The system according to claim 1, wherein: At least one feature includes printed markings on the ground.

4. The system according to claim 1, wherein: The control module is further configured to, with the processor, stitch the sensor data to form a scene of the environment and identify errors in features of the stitched sensor data, and wherein the video data is modified based on the identified errors.

5. The system according to claim 1, wherein: The control module is configured to modify the video data by smoothing the lines in the distorted ground plane.

6. The system according to claim 5, wherein: The control module is further configured to modify the video data by sharpening edges of smooth lines in the distorted ground plane.

7. The system according to claim 1, wherein: The control module is further configured to stitch the sensor data and the stored shapes to modify the video data.

8. A method for assisting an operator in operating a vehicle, comprising: receiving sensor data from a sensor system that senses an environment of the vehicle; Determining, by a processor, a scene of an environment based on sensor data; identifying, by a processor, coordinates of at least one feature within the scene; calculating, by a processor, a confidence score for the coordinates based on at least one of the vehicle speed data, the steering angle data, and the real-world geometry data; storing the coordinates as feature coordinates of at least one feature in the scene based on the confidence score, modifying, by a processor, the video data by warping a ground plane of one or more portions of the stitched scene data based on stored feature coordinates of features in the environment; as well as Display data is generated to display the modified video data for viewing by a vehicle operator.

Images