Vehicle image analysis system for peripheral camera

Abstract

A vehicle image analysis system includes one or more controllers that execute instructions to determine a source of image data that analyzes image data captured by one or more peripheral cameras. The one or more peripheral cameras are part of a vehicle occupant personal mobile device. The source of image data is an exterior environment around the vehicle or an interior environment representing an interior cabin of the vehicle.

Description

Technical Field

[0001] The present disclosure relates to a vehicle image analysis system that analyzes image data captured by one or more peripheral cameras. More specifically, the image analysis system determines an image data source, where the source is an external environment around the vehicle or an internal environment representing the vehicle's interior cabin. Background Art

[0002] Autonomous vehicles perform various tasks such as, but not limited to, perception, localization, mapping, path planning, decision making, and motion control. For example, an autonomous vehicle may include perception sensors such as cameras, lidar, and radar for collecting perception data about the vehicle's surrounding environment. It should be understood that, in addition to the autonomous driving system, other vehicle systems may also use the perception data, such as a lane change assist device. However, in some cases, one or more perception sensors such as cameras may not function due to various reasons.

[0003] In response to detecting that an in-vehicle camera is no longer functioning, a vehicle occupant can connect their personal mobile device, including a camera, to the vehicle's autonomous driving system via a wireless connection. The occupant's mobile device is sometimes referred to as a bring-your-own-device (BYOD) camera. Although BYOD cameras have various advantages, they may also pose some unique challenges. For example, an occupant may misposition the lens of the BYOD camera, thereby capturing image data representing the interior cabin rather than the external environment. Therefore, various vehicle systems that rely on the image data, such as the vehicle's autonomous driving system, need to first execute an algorithm to confirm that the image data captured by the BYOD camera represents the external environment around the vehicle. However, it should be understood that existing algorithms for determining whether the image data captured by a BYOD camera represents the external environment require a large amount of computation. Additionally, it should be understood that some vehicles may not include a processor with computing resources to determine whether the image data captured by a BYOD camera represents the external environment.

[0004] Therefore, although current systems achieve their intended purposes, there is still a need in the art for an improved method to determine whether a BYOD camera captures image data representing the external environment around the vehicle. Summary of the Invention

[0005] According to several aspects, the present invention discloses an image analysis system for a vehicle, which analyzes image data captured by one or more peripheral cameras. The image analysis system includes one or more controllers, wherein each of the one or more controllers includes one or more processors, and the one or more processors execute instructions to receive image data captured by one or more peripheral cameras. The one or more controllers classify each image frame of the image data captured by the one or more peripheral cameras within a predefined time period as a key frame or an incremental frame. The one or more controllers compare the average frame size of all incremental frames that are part of the image data captured by the one or more peripheral cameras within a predefined time period with a threshold average incremental frame size. The one or more controllers determine the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras, and the one or more peripheral cameras have a threshold pixel value difference, wherein the two image frames are separated by a time interval. In response to determining that the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras is less than or equal to the threshold pixel value difference, and determining that the average frame size of all incremental frames that are part of the image data captured by the one or more peripheral cameras within a predetermined time period is less than or equal to the threshold average incremental frame size, the one or more controllers determine that the image data captured by the one or more peripheral cameras represents the interior compartment of the vehicle. In response to determining that the image data captured by the one or more peripheral cameras represents the interior compartment of the vehicle, the one or more controllers instruct one or more vehicle systems to ignore the image data captured by the peripheral cameras.

[0006] In another aspect, in response to determining that the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras is greater than the threshold pixel value difference, or the average frame size of all incremental frames that are part of the image data captured by the one or more peripheral cameras within a predetermined time period is greater than the threshold average incremental frame size, the one or more controllers determine that the image data captured by the one or more peripheral cameras may represent the external environment around the vehicle.

[0007] In yet another aspect, in response to determining that the image data captured by the one or more peripheral cameras may represent the external environment around the vehicle, the one or more controllers determine a plurality of motion vectors related to the scene represented between two image frames that are part of the image data captured by the one or more peripheral cameras based on one or more video compression algorithms.

[0008] In one aspect, one or more processors of one or more controllers execute instructions to analyze the magnitude of each of a plurality of motion vectors between two image frames to determine a motion change corresponding to each motion vector, and to map one or more high-motion regions and one or more low-motion regions within the two image frames by comparing the motion change corresponding to each motion vector of the two image frames with a threshold motion value.

[0009] In another aspect, one or more processors of one or more controllers execute instructions to determine a percentage of two image frames that contain a high-motion region, compare the percentage of two image frames that contain a high-motion region with a threshold percentage, in response to determining that the percentage of two image frames that contain a high-motion region is at least equal to the threshold percentage, determine that image data captured by one or more peripheral cameras may represent the external environment around the vehicle, and in response to determining that the percentage of two image frames that contain a high-motion region is less than the threshold percentage, determine that image data captured by one or more peripheral cameras may represent the interior cabin of the vehicle.

[0010] In yet another aspect, one or more processors of one or more controllers execute instructions to analyze each macroblock of each image frame that is part of a plurality of image frames of image data captured by one or more peripheral cameras to determine a rising motion region within the plurality of image frames, wherein the plurality of image frames are collected over a period of time, compare the area size of the region representing the rising motion region within the plurality of image frames with a threshold coverage area, and in response to determining that the area size of the region representing the rising motion region is greater than or equal to the threshold coverage area, determine that image data captured by one or more peripheral cameras represents the external environment around the vehicle.

[0011] In one aspect, in response to determining that image data captured by one or more peripheral cameras represents the external environment around the vehicle, one or more controllers transmit the image data captured by one or more peripheral cameras to one or more vehicle systems.

[0012] In another aspect, in response to determining that image data captured by one or more peripheral cameras represents the external environment around the vehicle, one or more controllers execute one or more edge detection algorithms to identify ground markings set along the road, and in response to identifying lane markings within the image data captured by one or more peripheral cameras, confirm that the image data represents the external environment around the vehicle.

[0013] In yet another aspect, one or more peripheral cameras communicate electronically with one or more controllers, wherein the one or more peripheral cameras are part of a vehicle occupant personal mobile device.

[0014] In one aspect, in response to determining that image data captured by one or more peripheral cameras represents the interior cabin of a vehicle, one or more controllers direct one or more notification devices to generate a notification indicating that one or more peripheral cameras are capturing image data of the interior cabin of the vehicle.

[0015] In another aspect, the one or more peripheral cameras are bring-your-own-device (BYOD) cameras.

[0016] In yet another aspect, the one or more peripheral cameras represent devices that are temporarily connected to one or more controllers in response to determining that one or more in-vehicle cameras are inoperable.

[0017] In one aspect, a threshold average incremental frame size is selected to be at least equal to the average incremental frame size of an image data stream representing the interior cabin of the vehicle and less than the average incremental frame size of an image data stream representing the external environment around the vehicle.

[0018] In another aspect, the present disclosure provides a method for analyzing image data captured by one or more peripheral cameras of a vehicle. The method includes one or more controllers receiving image data captured by one or more peripheral cameras. The method further includes one or more controllers classifying each image frame of the image data captured by one or more peripheral cameras over a predefined time period as a key frame or an incremental frame. The method further includes one or more controllers comparing the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras over a predefined time period with a threshold average incremental frame size. The method further includes one or more controllers determining a pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras, the one or more peripheral cameras having a threshold pixel value difference, wherein the two image frames are separated by a time interval. In response to determining that the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras is less than or equal to the threshold pixel value difference, and in response to determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras over a predefined time period is less than or equal to the threshold average incremental frame size, the method includes determining that the image data captured by one or more peripheral cameras represents the interior cabin of the vehicle. In response to determining that the image data captured by one or more peripheral cameras represents the interior cabin of the vehicle, the method includes directing one or more vehicle systems to ignore the image data captured by the peripheral cameras.

[0019] In another aspect, in response to determining that the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras is greater than a threshold pixel value difference, or that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras within a predetermined time period is greater than a threshold average incremental frame size, the method includes determining that the image data captured by one or more peripheral cameras may represent the external environment around the vehicle.

[0020] In yet another aspect, in response to determining that the image data captured by one or more peripheral cameras may represent the external environment around the vehicle, the method includes determining a plurality of motion vectors associated with the scene represented between two image frames that are part of the aforementioned image data captured by one or more peripheral cameras based on one or more video compression algorithms.

[0021] In one aspect, the method includes analyzing the magnitude of each of the plurality of motion vectors between two image frames to determine the motion change corresponding to each motion vector, and mapping one or more high-motion regions and one or more low-motion regions within the two image frames by comparing the motion change corresponding to each motion vector of the two image frames with a threshold motion value.

[0022] In another aspect, the method includes determining the percentage of two image frames that contain high-motion regions, comparing the percentage of two image frames that contain high-motion regions with a threshold percentage, in response to determining that the percentage of two image frames that contain high-motion regions is at least equal to the threshold percentage, determining that the image data captured by one or more peripheral cameras may represent the external environment around the vehicle, and in response to determining that the percentage of two image frames that contain high-motion regions is less than the threshold percentage, determining that the image data captured by one or more peripheral cameras may represent the interior cabin of the vehicle.

[0023] In yet another aspect, the method includes analyzing each macroblock of each image frame that is part of a plurality of image frames of the image data captured by one or more peripheral cameras to determine the rising motion regions within the plurality of image frames, where the plurality of image frames are collected over a period of time, comparing the area size of the regions representing the rising motion regions within the plurality of image frames with a threshold coverage area, and in response to determining that the area size of the regions representing the rising motion regions is greater than or equal to the threshold coverage area, determining that the image data captured by one or more peripheral cameras represents the external environment around the vehicle.

[0024] In one aspect, the present invention discloses an image analysis system for a vehicle, which analyzes image data. The image analysis system includes one or more peripheral cameras that capture image data, one or more notification devices that generate notifications for vehicle occupants, and one or more controllers that are electronically communicable with the one or more peripheral cameras and the one or more notification devices. Each of the one or more controllers includes one or more processors that execute instructions to receive image data captured by the one or more peripheral cameras. The one or more controllers classify each image frame of the image data captured by the one or more peripheral cameras within a predefined time period as a key frame or an incremental frame. The one or more controllers compare the average frame size of all the incremental frames that are part of the image data captured by the one or more peripheral cameras within a predefined time period with a threshold average incremental frame size. The one or more controllers determine the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras, and the one or more peripheral cameras have a threshold pixel value difference, wherein the two image frames are separated by a time interval. In response to determining that the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras is less than or equal to the threshold pixel value difference, and determining that the average frame size of all the incremental frames that are part of the image data captured by the one or more peripheral cameras within a predetermined time period is less than or equal to the threshold average incremental frame size, the one or more controllers determine that the image data captured by the one or more peripheral cameras represents the interior compartment of the vehicle. Finally, in response to determining that the image data captured by the one or more peripheral cameras represents the interior compartment of the vehicle, the one or more controllers instruct the one or more notification devices to generate a notification indicating the image data of the interior compartment of the vehicle that the one or more peripheral cameras are capturing.

[0025] Further application areas will become apparent from the description provided herein. It should be understood that these descriptions and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way.

[0027] Figure 1 A schematic diagram of a vehicle including the disclosed image analysis system according to an exemplary embodiment is shown, the image analysis system including one or more controllers connected to one or more peripheral cameras;

[0028] Figure 2 is a block diagram showing Figure 1 the software architecture of the one or more controllers shown according to an exemplary embodiment;

[0029] Figure 3 is an exemplary illustration of an image frame that is part of image data captured by one or more peripheral cameras as shown by Figure 1 ; and

[0030] Figure 4 is a process flow diagram showing a method of analyzing image data captured by one or more peripheral cameras by the disclosed image analysis system as shown by Figure 1 . DETAILED DESCRIPTION

[0031] The following description is merely exemplary in nature and is not intended to limit the disclosure, application, or uses.

[0032] Referring to Figure 1 , a vehicle 10 including the disclosed image analysis system 12 is shown. As described below, the image analysis system 12 analyzes image data captured by one or more peripheral cameras 24 to determine whether the image data represents the external environment 14 around the vehicle 10 or the internal environment of the interior cabin 16 of the vehicle 10. The image analysis system 12 includes one or more controllers 20 that are in electronic communication with a plurality of sensing sensors 22 and one or more peripheral cameras 24. The plurality of sensing sensors 22 are configured to collect sensing data indicative of the external environment 14 around the vehicle 10. In a non-limiting embodiment as shown by Figure 1 , the plurality of sensing sensors 22 include one or more on-vehicle cameras 30, an inertial measurement unit (IMU) 32, a global positioning system (GPS) 34, radar 36, and lidar 38. However, it should be understood that additional sensors may also be used. It should be understood that the vehicle 10 can be any type of vehicle, such as, but not limited to, a sedan, a truck, a sport utility vehicle, a van, or a recreational vehicle.

[0033] One or more peripheral cameras 24 represent devices that are not normally connected to the one or more controllers 20 of the image analysis system 12. Instead, one or more peripheral cameras 24 represent devices that are temporarily connected to the one or more controllers 20 in response to determining that one or more on-vehicle cameras 30 are inoperable. Instead, one or more on-vehicle cameras 30 are part of the vehicle 10 and are normally connected to the one or more controllers 20 of the image analysis system 12. However, in some cases, one or more on-vehicle cameras 30 may become inoperable and no longer be able to capture data representing the external environment 14. Instead, an occupant 40 can electrically connect one or more peripheral cameras 24 to the one or more controllers 20. In a non-limiting embodiment as shown by Figure 1 , one or more peripheral cameras 24 are connected to the one or more controllers 20 in a wireless connection. However, it should be understood that a wired connection may also be used.

[0034] In one embodiment, one or more peripheral cameras 24 may be part of the personal mobile device of an occupant 40 of the vehicle 10. Some examples of personal mobile devices include, but are not limited to, smartphones, smartwatches, or tablets. For example, in one embodiment, one or more peripheral cameras 24 include bring-your-own-device (BYOD) cameras.

[0035] One or more vehicle systems 26 may be any type of vehicle system that requires image data of the external environment 14 to perform one or more tasks. Some examples of one or more vehicle systems 26 include, but are not limited to, an autonomous driving system and a lane change assist device. By way of example only, if one or more vehicle systems 26 is an autonomous driving system, the image data may be used for tasks such as, but not limited to, perception, localization, mapping, path planning, decision making, and motion control. As another example, if one or more vehicle systems 26 includes a lane change assist device, the image data may be used to determine whether the vehicle 10 can change lanes. It should be understood that the image analysis system 12 may analyze, in addition to analyzing the image data captured by one or more peripheral cameras 24 to determine whether the image data represents the external environment 14 or the internal environment, the image data for additional requirements specific to a particular vehicle system 26.

[0036] One or more notification devices 28 may include any device placed within the interior compartment 16 of the vehicle 10 for generating a notification to the occupant 40. The notification to the occupant may be a visual notification, an audio notification, or a tactile notification. In the non-limiting embodiment as Figure 1 shown, the notification device 28 is a display for showing text and graphics. In another embodiment, one or more notification devices 28 include a speaker for generating an audio notification. In yet another embodiment, one or more notification devices 28 include a smart seat system that includes one or more tactile devices mounted within the seat.

[0037] As described below, one or more controllers 20 of the image analysis system 12 determine an image data source captured by one or more peripheral cameras 24, where the source is the external environment 14 around the vehicle 10 or the internal environment representing the interior compartment 16 of the vehicle 10. It should be understood that if the image data source captured by one or more peripheral cameras 24 represents the internal environment of the interior compartment 16 of the vehicle 10, the image data may not be used by one or more vehicle systems 26 since one or more vehicle systems 26 require image data representing the external environment 14. In response to determining that the image data captured by one or more peripheral cameras 24 is image data representing the internal environment of the interior compartment 16 of the vehicle 10, in a non-limiting embodiment, one or more controllers 20 instruct one or more notification devices 28 to generate a notification to the occupant 40 indicating that one or more peripheral cameras 24 are capturing image data representing the internal environment of the interior compartment 16 of the vehicle 10. Otherwise, the image data captured by one or more peripheral cameras 24 is transmitted to one or more vehicle systems 26.

[0038] Figure 2 is a block diagram showing the software architecture of one or more controllers 20. One or more controllers 20 include a frame analysis module 50, a frame difference module 52, a spatial motion vector analysis module 54, a spatial and temporal motion vector analysis module 56, an edge detection module 58, a re-evaluation module 60, and a notification module 62. As Figure 2 shown, the frame analysis module 50 of one or more controllers 20 receives the image data captured by one or more peripheral cameras 24 and one or more vehicle dynamic parameters 64 from another external controller 66 that is part of the vehicle 10. One or more vehicle dynamic parameters 64 indicate the movement rate of the vehicle 10 and include parameters such as, but not limited to, vehicle speed.

[0039] The frame analysis module 50 of one or more controllers 20 classifies each image frame that is part of the image data captured by one or more peripheral cameras 24 within a predefined time period as a key frame or a non-key frame, where the non-key frames are called delta frames. The frame analysis module 50 compares the average frame size of all the delta frames that are part of the image data captured by one or more peripheral cameras 24 within a predefined time period with a threshold average delta frame size. In response to determining that the average frame size of all the delta frames that are part of the image data captured by one or more peripheral cameras 24 within a predefined time period is less than or equal to the threshold average delta frame size, the frame analysis module 50 determines that the image data captured by one or more peripheral cameras 24 may represent the interior compartment 16 of the vehicle 10.

[0040] It should be understood that the key frames represent the entire image frames, while the incremental frames only include the portions of the image frames that change over time based on the corresponding key frames. The threshold average incremental frame size is selected to be at least equal to the average incremental frame size of the image data stream representing the interior compartment 16 of the vehicle 10 and less than the average incremental frame size of the image data stream representing the external environment 14 around the vehicle 10. It should be understood that the average incremental frame size of the image data stream representing the interior compartment 16 of the vehicle 10 is less than the average incremental frame size of the image data stream representing the external environment 14 around the vehicle 10. This is because the image data stream representing the interior compartment 16 of the vehicle 10 changes less when compared to the image data stream representing the external environment 14 around the vehicle 10. It should be understood that the average incremental frame size of the image data stream representing the external environment 14 around the vehicle 10 depends on the motion rate of the vehicle 10 indicated by one or more vehicle dynamic parameters 64.

[0041] In response to determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 within a predefined time period is less than or equal to the threshold average incremental frame size, the frame analysis module 50 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the interior compartment 16 of the vehicle 10. Otherwise, the frame analysis module 50 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10. The frame analysis module 50 transmits the image data captured by one or more peripheral cameras 24 to the frame difference module 52.

[0042] The frame difference module 52 of one or more controllers 20 determines the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras 24, and one or more peripheral cameras 24 have a threshold pixel value difference, wherein the two image frames are separated by a time interval. The time interval is determined based on the motion rate of the vehicle 10 indicated by one or more vehicle dynamic parameters 64. The time interval is selected to reflect the changes in the scenery of the external environment 14 that occur when the vehicle 10 is traveling at the motion rate indicated by one or more vehicle dynamic parameters 64. In a non-limiting embodiment, the time interval is about one second. However, it should be understood that other values may also be used.

[0043] In one embodiment, each pixel that is part of each of the two image frames is evaluated. However, in another embodiment, only a portion of the pixels that are part of the two image frames are evaluated. However, it should be understood that the same pixel positions are evaluated between the two image frames to ensure consistency. In one embodiment, pixel values are expressed based on one or more numerical values, where each numerical value represents one of the colors that is part of the corresponding color model. By way of example only, a pixel based on the red-green-blue (RGB) color model would be represented as red: 6, green: 250, blue: 7. It should also be understood that while the RGB color model is described, pixels can also be based on any other color model, such as the cyan, magenta, yellow, and black (CMYB) color model and the YUV color model, where Y represents luminance or brightness, U represents the blue projection, and V represents the red projection.

[0044] The threshold pixel value difference is at least equal to the difference in pixel values based on the image data stream representing the interior compartment 16 of the vehicle 10 and is less than the difference in pixel values based on the image data stream representing the external environment 14 around the vehicle 10. The threshold pixel value difference is expressed as one or more numerical values, where each numerical value corresponds to one of the colors of the corresponding color model. By way of example, if the image data acquired by the perimeter camera 24 is based on the RGB color model and one pixel is represented as red: 60, green: 250, blue: 17 and its subsequent pixel is represented as red: 10, green: 50, blue: 7, then the pixel value difference is represented as red: 50, green: 200, blue: 10. It should be understood that the difference in pixel values based on the image data stream representing the interior compartment 16 of the vehicle 10 is less than the difference in pixel values based on the image data stream representing the external environment 14 around the vehicle 10. As described above, this is because the image data stream representing the interior compartment 16 of the vehicle 10 changes less when compared to the image data stream representing the external environment 14 around the vehicle 10.

[0045] In response to determining that the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras 24 is less than or equal to a threshold pixel value difference, the frame difference module 52 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the interior cabin 16 of the vehicle 10. Specifically, if both the frame analysis module 50 and the frame difference module 52 determine that the image data captured by one or more peripheral cameras 24 may represent the interior cabin of the vehicle 10, one or more controllers 20 determine that the image data captured by the peripheral cameras 24 represents the interior cabin of the vehicle 10 and instruct one or more vehicle systems 26 to ignore the image data captured by the peripheral cameras 24. In a non-limiting embodiment, the frame difference module 52 may then send a signal 68 to the notification module 62 indicating that the image data captured by the peripheral cameras 24 represents the interior cabin of the vehicle 10. Then, the notification module 62 instructs one or more notification devices 28 to generate a notification to the occupant 40 indicating that one or more peripheral cameras 24 are capturing image data representing the interior environment of the interior cabin 16 of the vehicle 10.

[0046] In response to the frame difference module 52 determining that the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras 24 is greater than the threshold pixel value difference, or the frame analysis module 50 determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 within a predefined time period is greater than a threshold average incremental frame size, one or more controllers 20 determine that the image data captured by one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10. Then, the frame difference module 52 may transmit the image data captured by one or more peripheral cameras 24 to the spatial motion vector analysis module 54 for spatial analysis.

[0047] The spatial motion vector analysis module 54 of one or more controllers 20 determines a plurality of motion vectors 70 associated with the scene represented between two image frames ([ Figure 3 shown), where the two image frames are part of the image data captured by one or more peripheral cameras 24. Some examples of video compression algorithms include, but are not limited to, AOMedia Video 1 (AV1) and Advanced Video Coding (AVC) (also known as H.264). Figure 3 is an exemplary illustration of one of the two image frames, where the image data captures a portion of the dashboard 80 of the vehicle 10 and the external environment 14 around the vehicle 10. Specifically, in Figure 3 the example shown, the external environment 14 around the vehicle 10 includes vegetation 82 with several trees on the right, the sky 84, and a grassy area 86 on the left.

[0048] ReferenceFigure 2 and Figure 3 The spatial motion vector analysis module 54 of one or more controllers 20 analyzes the magnitude of each of the plurality of motion vectors 70 between two image frames to determine the motion change corresponding to each motion vector 70. The spatial motion vector analysis module 54 of one or more controllers 20 maps one or more high-motion regions 90 and one or more low-motion regions 92 within the two image frames by comparing the motion change corresponding to each motion vector 70 of the two image frames with a threshold motion value. The threshold motion value depends on the motion rate of the vehicle 10 indicated by one or more vehicle dynamic parameters 64.

[0049] The threshold motion value represents the motion change experienced between two image frames when the corresponding image data represents the interior compartment 16 of the vehicle 10. If the motion change of the motion vector 70 is greater than the threshold motion value, the motion vector 70 corresponds to image data representing the external environment 14 around the vehicle 10. In the example shown in Figure 3 both the vegetation 82 and the grassy area 86 represent high-motion regions 90, while the dashboard 80 and the sky 84 located within the interior compartment 16 both represent low-motion regions 92. The spatial motion vector analysis module 54 of one or more controllers 20 determines the percentage of the two image frames that contain the high-motion region 90 and compares the percentage of the two image frames that contain the high-motion region 90 with a percentage threshold. The threshold percentage indicates that an important part of the scene presented between the two image frames captures the external environment 14 around the vehicle 10. In an embodiment, the threshold percentage is approximately twenty percent; however, other values may also be used.

[0050] In response to determining that the percentage of the two image frames that contain the high-motion region 90 is at least equal to the threshold percentage, the spatial motion vector analysis module 54 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10. Otherwise, the spatial motion vector analysis module 54 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the interior compartment 16 of the vehicle 10.

[0051] The spatial and temporal motion vector analysis module 56 of one or more controllers 20 analyzes each macroblock of each image frame that is part of a plurality of image frames of image data captured by one or more peripheral cameras 24 to determine an upward motion region within the plurality of image frames, where the plurality of image frames are collected over a period of time. The spatial and temporal motion vector analysis module 56 of one or more controllers 20 compares the region size representing the upward motion region within the plurality of image frames with a threshold coverage area. In response to determining that the region size representing the upward motion region is greater than or equal to the threshold coverage area, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 represents the external environment 14 around the vehicle 10.

[0052] The period of time is determined based on the motion rate of the vehicle 10 indicated by one or more vehicle dynamic parameters 64, the target confidence level, and the computing power of one or more controllers 20. In a non-limiting embodiment, the period of time is approximately ten seconds. The threshold coverage area is selected to exclude outlier data and to ignore trivial regions of the plurality of image frames that include portions of the external environment 14 and that cover less than five percent of the total area of the image frame. In a non-limiting embodiment, the threshold coverage area is approximately twenty percent of the total area of the image frame.

[0053] Now, the determination of the upward motion region within the plurality of image frames by the spatial and temporal motion vector analysis module 56 of one or more controllers 20 will be described. The spatial and temporal motion vector analysis module 56 of one or more controllers 20 first calculates a two-dimensional vector for each pixel that is part of each macroblock of each image frame that is part of the plurality of image frames. For example, in one embodiment, the size of the macroblock is determined to include 256 pixels. However, it should be understood that the macroblock can also include a different number of pixels. Then, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 adds together all of the two-dimensional vectors corresponding to each pixel that is part of a particular macroblock. The spatial and temporal motion vector analysis module 56 of one or more controllers 20 continues to add together all of the two-dimensional vectors corresponding to each pixel that is part of a particular macroblock until the end of the period of time.

[0054] In response to determining that a period of time has ended, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 determines the actual length of the overall motion vector corresponding to a particular macroblock based on the Pythagorean theorem formula. The spatial and temporal motion vector analysis module 56 of one or more controllers 20 compares the actual length of the overall motion vector corresponding to a particular macroblock with a threshold overall motion vector value. The threshold overall motion vector value is selected to represent motion observed within the external environment 14 around the vehicle 10. In response to determining that the actual length of the overall motion vector corresponding to a particular macroblock is greater than or equal to the threshold overall motion vector value, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 determines that the particular macroblock is part of an upward motion region within a plurality of image frames.

[0055] It should be understood that in the described embodiments, one or more vehicle systems 26 require image data of the external environment 14 to perform one or more tasks. Thus, in response to determining that the image data captured by one or more peripheral cameras 24 represents the external environment 14 around the vehicle 10, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 transmits the image data captured by one or more peripheral cameras 24 to one or more vehicle systems 26. Otherwise, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 represents the interior cabin 16 of the vehicle 10 and instructs one or more vehicle systems 26 to ignore the image data. In a non-limiting embodiment, the spatial and temporal motion vector analysis module 56 transmits a signal 94 to the notification module 62 indicating that the image data captured by the peripheral camera 24 represents the interior cabin of the vehicle 10. The notification module 62 then instructs one or more notification devices 28 to generate a notification to the occupant 40 indicating that one or more peripheral cameras 24 are capturing image data representing the interior environment of the interior cabin 16 of the vehicle 10.

[0056] In an alternative embodiment, one or more vehicle systems 26 may require image data representative of the interior cabin 16 of the vehicle 10 rather than the external environment 14 to perform one or more tasks. Accordingly, in response to determining that the image data captured by one or more perimeter cameras 24 represents the interior cabin 16 of the vehicle 10, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 transmits the image data captured by one or more perimeter cameras 24 to one or more vehicle systems 26. Otherwise, the spatial and temporal motion vector analysis module 56 of one or more controllers 20 determines that the image data captured by one or more perimeter cameras 24 represents the external environment 14 and instructs one or more vehicle systems 26 to ignore the image data. In one embodiment, the spatial and temporal motion vector analysis module 56 may transmit a signal 94 to the notification module 62, which then instructs one or more notification devices 28 to generate a notification to the occupant 40 indicating that one or more perimeter cameras 24 are capturing image data of the external environment 14.

[0057] In a non-limiting embodiment, the edge detection module 58 of one or more controllers 20 confirms that the image data captured by one or more perimeter cameras 24 represents the external environment 14 around the vehicle 10. Specifically, the edge detection module 58 of one or more controllers 20 performs one or more edge detection algorithms to identify lane markings disposed along the road in the image data captured by one or more perimeter cameras 24. In response to identifying lane markings within the image data captured by one or more perimeter cameras 24, the edge detection module 58 of one or more controllers 20 confirms that the image data captured by one or more perimeter cameras 24 represents the external environment 14. It should be understood that confirming the image data captured by one or more perimeter cameras 24 is optional and may be omitted in some embodiments.

[0058] In one embodiment, the re-evaluation module 60 of one or more controllers 20 may continuously monitor the azimuth angle θ and position d of one or more perimeter cameras 24 based on an ultra-wideband (UWB) sensor network 96. Refer Figure 1 and Figure 2 , the UWB sensor network 96 includes three or more anchors 98 mounted to the vehicle 10 ( Figure 1 shown), while the tags 100 ( Figure 1 ) of the UWB sensor network 96 are mounted to one or more perimeter cameras 24. The tags 100 are mobile sensors that are movably away from the vehicle 10 that transmits and receives sensor signals. Each anchor 98 of the UWB sensor network 96 wirelessly communicates with the tag 100 to transmit and receive sensor signals to track the tag 100.

[0059] The re-evaluation module 60 continuously monitors the azimuth angle θ and the position d of one or more peripheral cameras 24 of the UWB sensor network 96 until a change in the azimuth angle θ, the position d, or both the azimuth angle θ and the position d of one or more peripheral cameras 24 has exceeded a threshold. In response to determining that a change in the azimuth angle θ, the position d, or both the azimuth angle θ and the position d of one or more peripheral cameras 24 has exceeded the threshold, the re-evaluation module 60 instructs one or more controllers 20 to re-evaluate whether the image data captured by one or more peripheral cameras 24 represents the interior cabin 16 of the vehicle 10. The threshold indicates that the occupant 40 ( Figure 1 ) relocates one or more peripheral cameras 24 to capture image data of another environment. For example, although one or more peripheral cameras 24 may have been positioned to capture image data of the external environment 14, the occupant 40 may decide to relocate one or more peripheral cameras 24 to capture image data representing the interior cabin 16.

[0060] Figure 4 is a process flow diagram of a method 400 for analyzing image data captured by one or more peripheral cameras 24 by the Figure 1 shown disclosed image analysis system 12. Generally referring to Figure 1 、 Figure 2 and Figure 4 , the method 400 can start at decision block 402. In decision block 402, one or more controllers 20 continue to monitor one or more peripheral cameras 24 until image data captured by one or more peripheral cameras 24 is received. In response to receiving the image data, the method 400 proceeds to block 404.

[0061] In block 404, the frame analysis module 50 of one or more controllers 20 classifies each image frame of the image data captured by one or more peripheral cameras 24 within a predefined time period as a key frame or an incremental frame. Then, the method 400 can proceed to block 406.

[0062] In block 406, the frame analysis module 50 of one or more controllers 20 compares the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 over a predefined time period with a threshold average incremental frame size. In response to determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 over a predefined time period is less than or equal to the threshold average incremental frame size, the frame analysis module 50 determines that the image data captured by one or more peripheral cameras 24 may represent the interior compartment 16 of the vehicle 10. In response to determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 over a predefined time period is greater than the threshold average incremental frame size, the frame analysis module 50 determines that the image data captured by one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10. Then, method 400 can proceed to block 408.

[0063] In block 408, the frame difference module 52 of one or more controllers 20 determines the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras, where the one or more peripheral cameras have a threshold pixel value difference, and where the two image frames are separated by a time interval. Then, method 400 can proceed to decision block 410.

[0064] In decision block 412, in response to the frame difference module 52 of one or more controllers 20 determining that the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras 24 is less than or equal to the threshold pixel value difference, and in response to the frame difference module 52 of one or more controllers 20 determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 over a predefined time period is less than or equal to the threshold average incremental frame size, the frame difference module 52 determines that the image data captured by one or more peripheral cameras represents the interior compartment of the vehicle 10. Then, the frame difference module 52 instructs one or more vehicle systems 26 to ignore the image data captured by the peripheral cameras 24. Then, method 400 can proceed to block 412.

[0065] In block 412, in response to determining that the image data captured by one or more peripheral cameras 24 represents the interior compartment 16 of the vehicle 10, the frame difference module 52 sends a signal 68 to the notification module 62 indicating that the image data captured by the peripheral cameras 24 represents the interior compartment of the vehicle 10. Then, the notification module 62 instructs one or more notification devices 28 to generate a notification to the occupant 40 indicating that one or more peripheral cameras 24 are capturing image data of the interior environment representing the interior compartment 16 of the vehicle 10. It should be understood that in an embodiment, the notification to the occupant 40 may not be sent. Then, method 400 can terminate.

[0066] Return to block 410. In response to the frame difference module 52 of one or more controllers 20 determining that the pixel value difference between two image frames that are part of the image data captured by one or more peripheral cameras 24 is greater than a threshold pixel value difference, or the frame difference module 52 of one or more controllers 20 determining that the average frame size of all incremental frames that are part of the image data captured by one or more peripheral cameras 24 over a predetermined time period is greater than a threshold average incremental frame size, the frame difference module 52 determines that the image data captured by one or more peripheral cameras may represent the external environment 14 around the vehicle 10, and method 400 proceeds to block 414.

[0067] In block 414, in response to determining that the image data captured by one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10, the spatial motion vector analysis module 54 of one or more controllers 20 determines a plurality of motion vectors 70 associated with the scene represented between two image frames that are part of the image data captured by one or more peripheral cameras 24 based on one or more video compression algorithms ( Figure 3 as shown). Then, method 400 may proceed to block 416.

[0068] In block 416, the spatial motion vector analysis module 54 of one or more controllers 20 analyzes the magnitude of each of the plurality of motion vectors 70 ( Figure 3 as shown) between two image frames to determine the motion change corresponding to each motion vector 70, and maps one or more high-motion regions 90 and one or more low-motion regions 92 within the two image frames by comparing the motion change corresponding to each motion vector 70 of the two image frames with a threshold motion value. Then, method 400 may proceed to block 418.

[0069] In block 418, the spatial motion vector analysis module 54 of one or more controllers 20 determines the percentage of the two image frames that contain the high-motion region 90 and compares the percentage of the two image frames that contain the high-motion region 90 with a threshold percentage. In response to determining that the percentage of the two image frames that contain the high-motion region 90 is at least equal to the threshold percentage, the spatial motion vector analysis module 54 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10. In response to determining that the percentage of the two image frames that contain the high-motion region 90 is less than the threshold percentage, the spatial motion vector analysis module 54 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 may represent the interior compartment 16 of the vehicle 10. Then, method 400 may proceed to block 420.

[0070] In block 420, a spatial and temporal motion vector analysis module 56 of one or more controllers 20 analyzes each macroblock of each image frame that is part of a plurality of image frames of image data captured by one or more peripheral cameras 24 to determine an upward motion region within the plurality of image frames, where the plurality of image frames are collected over a period of time. Then, method 400 may proceed to decision block 422.

[0071] In decision block 422, a spatial and temporal motion vector analysis module 56 of one or more controllers 20 compares the area size of the region representing the upward motion region within the plurality of image frames with a threshold coverage area. In response to determining that the image data captured by one or more peripheral cameras 24 represents the interior compartment 16 of vehicle 10. Then, the spatial and temporal motion vector analysis module 56 instructs one or more vehicle systems 26 to ignore the image data captured by peripheral cameras 24. Then, method 400 may proceed to block 424.

[0072] In block 424, a spatial and temporal motion vector analysis module 56 of one or more controllers 20 transmits a signal 94 to a notification module 62 indicating that the image data captured by peripheral cameras 24 represents the interior compartment of vehicle 10. Then, notification module 62 instructs one or more notification devices 28 to generate a notification to occupant 40 indicating that one or more peripheral cameras 24 are capturing image data representing the interior environment of interior compartment 16 of vehicle 10. It should be understood that in an embodiment, the notification may not be sent to occupant 40. Then, method 400 may terminate.

[0073] Returning to decision block 422, in response to determining that the area size of the region representing the upward motion region is greater than or equal to the threshold coverage area, a spatial and temporal motion vector analysis module 56 of one or more controllers 20 determines that the image data captured by one or more peripheral cameras 24 represents the external environment 14 around vehicle 10. Then, method 400 may proceed to block 426.

[0074] In block 426, in response to determining that the image data captured by one or more peripheral cameras 24 represents the external environment 14 around vehicle 10, a spatial and temporal motion vector analysis module 56 of one or more controllers 20 transmits the image data captured by one or more peripheral cameras 24 to one or more vehicle systems 26. Then, method 400 may terminate.

[0075] Referring generally to the illustrated example, the disclosed image analysis system has various technical effects and benefits. Specifically, the disclosed image analysis system provides a relatively lightweight method for determining whether image data captured by a peripheral camera is suitable for use by one or more vehicle systems (such as an autonomous driving system). It should be understood that the disclosed method requires significantly fewer computing resources compared to various currently available computer vision-based methods.

[0076] A controller can refer to an electronic circuit, combinational logic circuit, field programmable gate array (FPGA), a processor (shared, dedicated, or a group) that executes code, or a combination of some or all of the above, such as in a system on a chip. Additionally, a controller can be microprocessor-based, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor can operate under the control of an operating system resident in the memory. The operating system can manage computer resources such that computer program code embodied as one or more computer software applications (such as applications resident in the memory) can have instructions executed by the processor. In an alternative embodiment, the processor can directly execute the application, in which case the operating system can be omitted.

[0077] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. An image analysis system for a vehicle, the image analysis system analyzing image data captured by one or more peripheral cameras, the image analysis system comprising: One or more controllers, wherein each of the one or more controllers includes one or more processors, and the one or more processors execute instructions to: Receive the image data captured by the one or more peripheral cameras; Classify each image frame of the image data captured by the one or more peripheral cameras within a predetermined time period as a key frame or an incremental frame; Compare the average frame size of all the incremental frames that are part of the image data captured by the one or more peripheral cameras within the predefined time period with a threshold average incremental frame size; Determine the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras, the one or more peripheral cameras having a threshold pixel value difference, wherein the two image frames are separated by a time interval; In response to determining that the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras is less than or equal to the threshold pixel value difference, and determining that the average frame size of all the incremental frames that are part of the image data captured by the one or more peripheral cameras within the predetermined time period is less than or equal to the threshold average incremental frame size, determine that the image data captured by the one or more peripheral cameras represents the interior cabin of the vehicle; and In response to determining that the image data captured by the one or more peripheral cameras represents the interior cabin of the vehicle, instruct one or more vehicle systems to ignore the image data captured by the peripheral cameras.

2. The image analysis system according to claim 1, wherein, The one or more processors of the one or more controllers execute instructions to: In response to determining that the pixel value difference between two image frames that are part of the image data captured by the one or more peripheral cameras is greater than the threshold pixel value difference, or the average frame size of all the incremental frames that are part of the image data captured by the one or more peripheral cameras within the predetermined time period is greater than the threshold average incremental frame size, determine that the image data captured by the one or more peripheral cameras may represent the external environment around the vehicle.

3. The image analysis system according to claim 2, wherein, The one or more processors of the one or more controllers execute instructions to: In response to determining that the image data captured by the one or more peripheral cameras may represent the external environment around the vehicle, determine a plurality of motion vectors related to the scene represented between two image frames that are part of the image data captured by the one or more peripheral cameras based on one or more video compression algorithms.

4. The image analysis system according to claim 3, wherein, The one or more processors of the one or more controllers execute instructions to: Analyze the magnitude of each of the plurality of motion vectors between the two image frames to determine the motion change corresponding to each motion vector; And Map one or more high-motion regions and one or more low-motion regions within the two image frames by comparing the motion change corresponding to each motion vector of the two image frames with a threshold motion value.

5. The image analysis system according to claim 4, wherein, One or more processors of the one or more controllers execute instructions to: Determine a percentage of two image frames that include the high-motion region; Compare the percentage of two image frames that include the high-motion region with a threshold percentage; In response to determining that the percentage of two image frames that include the high-motion region is at least equal to the threshold percentage, determine that image data captured by the one or more peripheral cameras may represent the external environment around the vehicle; And In response to determining that the percentage of two image frames that include the high-motion region is less than the threshold percentage, determine that image data captured by the one or more peripheral cameras may represent the interior compartment of the vehicle.

6. The image analysis system according to claim 5, wherein, One or more processors of the one or more controllers execute instructions to: Analyze each macroblock of each image frame that is part of a plurality of image frames of image data captured by the one or more peripheral cameras to determine a rising motion region within the plurality of image frames, wherein the plurality of image frames are collected over a period of time; Compare a region size representing the rising motion region within the plurality of image frames with a threshold coverage area; and In response to determining that the region size representing the rising motion region is greater than or equal to the threshold coverage area, determine that image data captured by the one or more peripheral cameras represents the external environment around the vehicle.

7. The image analysis system according to claim 6, wherein, One or more processors of the one or more controllers execute instructions to: In response to determining that image data captured by the one or more peripheral cameras represents the external environment around the vehicle, transmit the image data captured by the one or more peripheral cameras to one or more vehicle systems.

8. The image analysis system according to claim 6, wherein, One or more processors of the one or more controllers execute instructions to: In response to determining that image data captured by the one or more peripheral cameras represents the external environment around the vehicle, execute one or more edge detection algorithms to identify ground markings disposed along a road; and In response to identifying lane markings within the image data captured by the one or more peripheral cameras, confirm that the image data represents the external environment around the vehicle.

9. The image analysis system according to claim 1, wherein, The one or more peripheral cameras communicate electronically with the one or more controllers, and wherein the one or more peripheral cameras are part of a personal mobile device of a vehicle occupant.

10. The image analysis system according to claim 1, wherein, One or more processors of the one or more controllers execute instructions to: In response to determining that image data captured by the one or more peripheral cameras represents the interior compartment of the vehicle, instruct one or more notification devices to generate a notification indicating that the one or more peripheral cameras are capturing image data of the interior compartment of the vehicle.

Images