Image analysis system for a vehicle peripheral camera

Abstract

An image analysis system for a vehicle that analyzes image data acquired by one or more peripheral cameras includes one or more controllers that execute instructions to determine a source of the image data. The one or more peripheral cameras are part of a vehicle occupant's personal mobile device. The source of the image data is either an external environment surrounding the vehicle or an internal environment representative of an interior of the vehicle.

Description

introduction

[0001] The present disclosure relates to an image analysis system for a vehicle that analyzes image data acquired by one or more peripheral cameras. In particular, the image analysis system determines a source of the image data, wherein the source is either an external environment surrounding the vehicle or an internal environment representing an interior of the vehicle.

[0002] An autonomous vehicle performs various tasks such as perception, localization, mapping, path planning, decision-making, and motion control. For example, an autonomous vehicle may contain perception sensors such as a camera, LiDAR, and radar to collect perception data about the environment around the vehicle. It should be understood that the perception data may be used by vehicle systems other than an autonomous driving system, such as a lane change assistant. However, in some cases, one or more perception sensors, such as the camera, may become inoperable for various reasons.

[0003] In response to discovering that the in-vehicle camera is no longer functioning, a vehicle occupant can wirelessly connect their personal mobile device containing a camera to the vehicle's autonomous driving system. The occupant's mobile device is sometimes referred to as a bring-your-own-device (BYOD) camera. While BYOD cameras offer some advantages, they can also present some unique challenges. For example, the occupant may misposition the BYOD camera lens, resulting in image data representative of the interior rather than the exterior environment.Accordingly, various vehicle systems that rely on image data, such as the vehicle's autonomous driving system, are required to first execute an algorithm to confirm that the image data captured by the BYOD camera is representative of the external environment around the vehicle. However, it should be noted that existing algorithms for determining whether the image data captured by the BYOD camera represents the external environment are computationally intensive. Furthermore, it should also be noted that some vehicles may not contain processors with the computing resources to determine whether the image data captured by the BYOD camera represents the external environment.

[0004] Thus, while current systems serve their intended purpose, there is a need in the art for an improved approach to determining whether a BYOD camera is capturing image data representative of the external environment around the vehicle. Summary

[0005] According to several aspects, an image analysis system for a vehicle is disclosed that analyzes image data acquired by one or more peripheral cameras. The image analysis system includes one or more controllers, wherein the one or more controllers each include one or more processors that execute instructions to receive the image data acquired by the one or more peripheral cameras. The one or more controllers classify each image frame of the image data acquired by the one or more peripheral cameras over a predefined period of time as either a keyframe or a delta frame.The one or more controllers compare an average frame size of all the delta frames included in the image data acquired by the one or more peripheral cameras over the predefined period of time to a threshold average delta frame size. The one or more controllers determine a difference in pixel values between two image frames included in the image data acquired by the one or more peripheral cameras to a threshold pixel value difference, where the two image frames are separated by a time interval.In response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is less than or equal to the pixel value difference threshold, and determining that the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is less than or equal to the average delta frame size threshold, the one or more controllers determine that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle.In response to determining that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle, the one or more controllers instruct one or more vehicle systems to disregard or discard the image data acquired by the peripheral camera.

[0006] In another aspect, in response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is greater than the pixel value difference threshold or the average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is greater than the average delta frame size threshold, the one or more controllers determine that the image data acquired by the one or more peripheral cameras may represent an external environment around the vehicle.

[0007] In yet another aspect, in response to determining that the image data acquired by the one or more peripheral cameras may represent the external environment around the vehicle, the one or more controllers determine a plurality of motion vectors related to a scene represented between two image frames that are part of the image data acquired by the one or more peripheral cameras based on one or more video compression algorithms.

[0008] In one aspect, the one or more processors of the one or more controllers execute instructions to analyze a magnitude of each of the plurality of motion vectors between the two image frames to determine a change in motion corresponding to each motion vector, and to map one or more high motion regions and one or more low motion regions within the two image frames by comparing the change in motion corresponding to each motion vector of the two image frames to a threshold motion value.

[0009] In another aspect, the one or more processors of the one or more controllers execute instructions to determine a percentage of the two image frames containing the high-motion areas, compare the percentage of the two image frames containing the high-motion areas to a threshold percentage value, in response to determining that the percentage of the two image frames containing the high-motion areas is at least equal to the threshold percentage value, determine that the image data acquired via the one or more peripheral cameras may represent the external environment around the vehicle, and in response to determining that the percentage of the two image frames containing the high-motion areas is less than the threshold percentage value,that the image data captured by the one or more peripheral cameras may represent the interior of the vehicle.

[0010] In yet another aspect, the one or more processors of the one or more controllers execute instructions to analyze each macroblock of each image frame that is part of a plurality of image frames from the image data acquired by the one or more peripheral cameras to determine areas of increased motion within the plurality of image frames, the plurality of image frames being collected over a period of time, compare a size of an area representing the area of increased motion within the plurality of image frames to a coverage area threshold, and in response to determining that the size of the area representing the area of increased motion is equal to or greater than the coverage area threshold,that the image data captured by the one or more peripheral cameras represent the external environment around the vehicle.

[0011] In one aspect, in response to determining that the image data acquired by the one or more peripheral cameras represents the external environment around the vehicle, the one or more controllers transmit the image data acquired by the one or more peripheral cameras to one or more vehicle systems.

[0012] In another aspect, in response to determining that the image data acquired by the one or more peripheral cameras represents the external environment around the vehicle, the one or more controllers execute one or more edge detection algorithms to identify a lane marker located along a roadway and, in response to identifying lane markers within the image data acquired by the one or more peripheral cameras, confirm that the image data is representative of the external environment around the vehicle.

[0013] In yet another aspect, the one or more peripheral cameras are in electronic communication with the one or more controllers, and wherein the one or more peripheral cameras are part of a personal mobile device of an occupant of the vehicle.

[0014] In one aspect, in response to determining that the image data captured by the one or more peripheral cameras represents an interior of the vehicle, the one or more controllers instruct one or more notification devices to generate a notification indicating that the one or more peripheral cameras capture image data of the interior of the vehicle.

[0015] In another aspect, the one or more peripheral cameras are a bring-your-own-device (BYOD) camera.

[0016] In yet another aspect, the one or more peripheral cameras represent a device that is temporarily connected to the one or more controllers in response to determining that one or more on-board cameras are inoperative.

[0017] In one aspect, the average delta frame size threshold is selected to be at least equal to the average delta frame size of a stream of image data representative of an interior of the vehicle and less than an average delta frame size of a stream of image data representative of an external environment around the vehicle.

[0018] In another aspect, a method for analyzing image data acquired by one or more peripheral cameras of a vehicle is disclosed. The method includes receiving, by one or more controllers, the image data acquired by the one or more peripheral cameras. The method also includes classifying, by the one or more controllers, each image frame of the image data acquired by the one or more peripheral cameras over a predefined period of time as either a key frame or a delta frame. The method further includes comparing, by the one or more controllers, an average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time to a threshold value for the average delta frame size.The method also includes determining, by the one or more controllers, a difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras, with a threshold for the pixel value difference, wherein the two image frames are spaced apart by a time interval.In response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is less than or equal to the pixel value difference threshold, and determining that the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is less than or equal to the average delta frame size threshold, the method includes determining that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle.In response to determining that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle, the method includes instructing one or more vehicle systems to discard the image data acquired by the peripheral camera.

[0019] In another aspect, the method comprises, in response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is greater than the pixel value difference threshold or the average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is greater than the average delta frame size threshold, determining that the image data acquired by the one or more peripheral cameras may represent an external environment around the vehicle.

[0020] In yet another aspect, the method comprises, in response to determining that the image data acquired by the one or more peripheral cameras may represent the external environment around the vehicle, determining a plurality of motion vectors related to a scene represented between two image frames that are part of the image data acquired by the one or more peripheral cameras based on one or more video compression algorithms.

[0021] In one aspect, the method comprises analyzing a magnitude of each of the plurality of motion vectors between the two image frames to determine a change in motion corresponding to each motion vector, and mapping one or more high motion regions and one or more low motion regions within the two image frames by comparing the change in motion corresponding to each motion vector of the two image frames to a threshold motion value.

[0022] In another aspect, the method comprises determining a percentage of the two image frames containing the high-motion areas, comparing the percentage of the two image frames containing the high-motion areas to a threshold percentage value, in response to determining that the percentage of the two image frames containing the high-motion areas is at least equal to the threshold percentage value, determining that the image data acquired via the one or more peripheral cameras may represent the external environment surrounding the vehicle, and in response to determining that the percentage of the two image frames containing the high-motion areas is less than the threshold percentage value, determining that the image data acquired via the one or more peripheral cameras may represent the interior of the vehicle.

[0023] In yet another aspect, the method comprises analyzing each macroblock of each image frame that is part of a plurality of image frames from the image data acquired by the one or more peripheral cameras to determine areas of increased motion within the plurality of image frames, the plurality of image frames being collected over a period of time, comparing a size of an area representing the area of increased motion within the plurality of image frames to a coverage area threshold, and in response to determining that the size of the area representing the area of increased motion is equal to or greater than the coverage area threshold, determining that the image data acquired by the one or more peripheral cameras represents the external environment around the vehicle.

[0024] In one aspect, an image analysis system for a vehicle that analyzes image data is disclosed. The image analysis system includes one or more peripheral cameras that capture the image data, one or more notification devices that generate a notification for an occupant of the vehicle, and one or more controllers in electronic communication with the one or more peripheral cameras and the one or more notification devices. The one or more controllers each include one or more processors that execute instructions to receive the image data captured by the one or more peripheral cameras. The one or more controllers classify each image frame of the image data captured by the one or more peripheral cameras over a predefined period of time as either a key frame or a delta frame.The one or more controllers compare an average frame size of all the delta frames included in the image data acquired by the one or more peripheral cameras over the predefined period of time with a threshold for the average delta frame size. The one or more controllers determine a difference in pixel values between two image frames included in the image data acquired by the one or more peripheral cameras with a threshold for the pixel value difference, where the two image frames are spaced apart by a time interval.In response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is less than or equal to the pixel value difference threshold, and determining that the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is less than or equal to the average delta frame size threshold, the one or more controllers determine that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle.Finally, in response to determining that the image data captured by the one or more peripheral cameras represents an interior of the vehicle, the one or more controllers instruct the one or more notification devices to generate a notification indicating that the one or more peripheral cameras capture image data of the interior of the vehicle.

[0025] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Brief description of the drawings

[0026] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. 1 illustrates a schematic representation of a vehicle incorporating the disclosed image analysis system including one or more controllers connected to one or more peripheral cameras, according to an exemplary embodiment; Fig. 2 is a block diagram showing the software architecture of one or more controllers used in Fig. 1, according to an exemplary embodiment; Fig. 3 is an exemplary representation of a picture frame that is part of by means of the one or more, in Fig. 1, according to an exemplary embodiment; and Fig. 4 is a process flow diagram illustrating a method for analyzing the image data acquired by the one or more peripheral cameras by the Fig. 1 illustrates the disclosed image analysis system according to an exemplary embodiment. Detailed description

[0027] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0028] Referring to Fig. 1, a vehicle 10 is illustrated that includes the disclosed image analysis system 12. As explained below, the image analysis system 12 analyzes image data acquired by one or more peripheral cameras 24 to determine whether the image data represents an external environment 14 around the vehicle 10 or an internal environment representative of the interior 16 of the vehicle 10. The image analysis system 12 includes one or more controllers 20 in electronic communication with a plurality of perception sensors 22 and the one or more peripheral cameras 24. The plurality of perception sensors 22 are configured to acquire perception data indicative of the external environment 14 around the vehicle 10. In the non-limiting embodiment as shown in Fig. 1, the plurality of perception sensors 22 includes one or more on-board cameras 30, an inertial measurement unit (IMU) 32, a global positioning system (GPS) 34, radar 36, and LiDAR 38; however, it should be appreciated that additional sensors may also be used. It should be understood that the vehicle 10 may be any type of vehicle, such as, but not limited to, a sedan, a truck, an SUV, a van, or a motorhome.

[0029] The one or more peripheral cameras 24 represent a device that is not normally connected to the one or more controllers 20 of the image analysis system 12. Instead, the one or more peripheral cameras 24 represent a device that is temporarily connected to the one or more controllers 20 in response to determining that the one or more on-board cameras 30 are inoperative. In contrast, the one or more on-board cameras 30 are part of the vehicle 10 and are normally connected to the one or more controllers 20 of the image analysis system 12. However, in some cases, the one or more on-board cameras 30 may become inoperative and no longer able to collect data representative of the external environment 14.Instead, the occupant 40 may electronically connect the one or more peripheral cameras 24 to the one or more controllers 20. In the non-limiting embodiment shown in FIG. Fig. 1, the one or more peripheral cameras 24 are connected to the one or more controllers 20 based on a wireless connection; however, it should be understood that a wired connection may also be used.

[0030] In one embodiment, the one or more peripheral cameras 24 may be part of a personal mobile device of an occupant 40 of the vehicle 10. Some examples of personal mobile devices include, but are not limited to, a smartphone, a smartwatch, or a tablet computer. In one embodiment, the one or more peripheral cameras 24 include, for example, a bring-your-own-device (BYOD) camera.

[0031] The one or more vehicle systems 26 may be any type of vehicle system that requires image data of the external environment 14 to perform one or more tasks. Some examples of the one or more vehicle systems 26 include, but are not limited to, an autonomous driving system and a lane change assist system. If the one or more vehicle systems 26 is, for example only, an autonomous driving system, then the image data may be used for tasks such as, but not limited to, perception, localization, mapping, path planning, decision making, and motion control. As another example, if the one or more vehicle systems 26 include a lane change assist system, then the image data may be used to determine whether the vehicle 10 is able to change lanes.It should be understood that in addition to the image analysis system 12 analyzing the image data acquired by one or more peripheral cameras 24 to determine whether the image data represents an external environment 14 or the internal environment, the image data may also be analyzed with regard to additional requirements specific to a particular vehicle system 26.

[0032] The one or more notification devices 28 may comprise any device disposed within the interior 16 of the vehicle 10 to generate a notification directed to the occupant 40. The notification directed to the occupant may be a visual notification, an audible notification, or a haptic notification. In the non-limiting embodiment, as shown in Fig. 1, the notification device 28 is a display that displays text and graphics. In another embodiment, the one or more notification devices 28 include a speaker for generating an audible notification. In yet another embodiment, the one or more notification devices 28 include a smart seat system that includes one or more haptic devices installed in the seats.

[0033] As explained below, the one or more controllers 20 of the image analysis system 12 determine a source of the image data acquired by the one or more peripheral cameras 24, where the source is either the external environment 14 surrounding the vehicle 10 or the internal environment representative of the interior 16 of the vehicle 10. It should be understood that if the source of the image data acquired by the one or more peripheral cameras 24 is the internal environment representative of the interior 16 of the vehicle 10, then the image data cannot be used by the one or more vehicle systems 26 because the one or more vehicle systems 26 instead require image data representative of the external environment 14.In one non-limiting embodiment, in response to determining that the image data captured by the one or more peripheral cameras 24 relates to the interior environment representative of the interior 16 of the vehicle 10, the one or more controllers 20 instruct the one or more notification devices 28 to generate the notification to the occupant 40 indicating that the one or more peripheral cameras 24 are capturing image data of the interior environment representative of the interior 16 of the vehicle 10. Otherwise, the image data captured by the one or more peripheral cameras 24 is transmitted to the one or more vehicle systems 26.

[0034] Fig. 2 is a block diagram illustrating the software architecture of the one or more controllers 20. The one or more controllers 20 include a frame analysis module 50, a frame differentiation module 52, a spatial motion vector analysis module 54, a spatial and temporal motion vector analysis module 56, an edge detection module 58, a re-evaluation module 60, and a notification module 62. As shown in Fig. As shown in Figure 2, the frame analysis module 50 of the one or more controllers 20 receives the image data acquired by the one or more peripheral cameras 24 and one or more vehicle dynamic parameters 64 from another external controller 66 that is part of the vehicle 10. The one or more vehicle dynamic parameters 64 indicate a rate of movement of the vehicle 10 and include parameters such as, but not limited to, vehicle speed.

[0035] The frame analysis module 50 of the one or more controllers 20 classifies each image that is part of the image data acquired by the one or more peripheral cameras 24 over a predefined period of time as either a key frame or a non-key frame, with the non-key frame being referred to as a delta frame. The frame analysis module 50 compares an average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time to a threshold value for the average delta frame size.In response to determining that the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time is less than or equal to the average delta frame size threshold, the frame analysis module 50 determines that the image data acquired by the one or more peripheral cameras 24 may represent the interior 16 of the vehicle 10. .

[0036] It should be understood that a key frame represents the entire image frame, while the delta frame only contains portions of the image frame that have changed over time based on the particular key frame. The average delta frame size is chosen to be at least equal to an average delta frame size of a stream of image data representative of the interior 16 of the vehicle 10 and smaller than an average delta frame size of a stream of image data representative of the external environment 14 around the vehicle 10. It should be understood that the average delta frame size of the stream of image data representative of the interior 16 of the vehicle 10 is smaller than the average delta frame size of the stream of image data representative of the external environment 14 around the vehicle 10.This is because the stream of image data representative of the interior 16 of the vehicle 10 changes less than the stream of image data representative of the external environment 14 surrounding the vehicle 10. It should be appreciated that the average delta frame size of the stream of image data representative of the external environment 14 surrounding the vehicle 10 depends on the rate of motion of the vehicle 10, as indicated by the one or more vehicle dynamics parameters 64.

[0037] In response to determining that the average frame size of all delta frames included in the image data acquired by the one or more peripheral cameras 24 over the predefined period of time is less than or equal to the average delta frame size threshold, the frame analysis module 50 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 may represent the interior 16 of the vehicle 10. Otherwise, the frame analysis module 50 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 may represent the external environment 14 surrounding the vehicle 10. The frame analysis module 50 transmits the image data acquired by the one or more peripheral cameras 24 to the frame differentiation module 52.

[0038] The frame differentiation module 52 of the one or more controllers 20 determines a difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras 24, with a threshold for the pixel value difference, where the two image frames are spaced apart by a time interval. The time interval is determined based on the rate of movement of the vehicle 10 indicated by the one or more vehicle dynamics parameters 64. The time interval is selected to reflect a change in the scenery of the external environment 14 that occurs when the vehicle 10 is traveling at the rate of movement indicated by the one or more vehicle dynamics parameters 64. In one non-limiting embodiment, the time interval is approximately one second; however, it should be understood that other values may be used.

[0039] In one embodiment, each pixel that is part of each of the two image frames is evaluated; however, in another embodiment, only a portion of the pixels that are part of the two image frames are evaluated. It should be understood, however, that the same pixel position is evaluated between the two image frames to ensure consistency. In one embodiment, the pixel values are expressed based on one or more numerical values, where each numerical value represents one of the colors that are part of a corresponding color model. By way of example only, a pixel based on the red-green-blue (RGB) color model would be expressed as follows: Red: 6, Green: 250, Blue: 7. It should also be understood that, although the RGB color model is described, the pixels can be based on any other color model, such as the cyan, magenta, yellow, and black (CMYB) color model.: Cyan, Magenta, Yellow, Black) and the YUV color model, where Y stands for luma or brightness, U represents the blue projection and V represents the red projection.

[0040] The threshold for the pixel value difference is at least equal to a difference in pixel values based on the stream of image data representative of the interior 16 of the vehicle 10 and less than a difference in pixel values based on the stream of image data representative of the external environment 14 surrounding the vehicle 10. The threshold for the pixel value difference is expressed as one or more numerical values, each numerical value corresponding to one of the colors of the corresponding color model. For example, if the image data acquired by the peripheral camera 24 is based on the RGB color model, and if one of the pixels is expressed as Red: 60, Green: 250, Blue: 17 and the subsequent pixel is expressed as Red: 10, Green: 50, Blue: 7, then the difference in pixel values is expressed as Red: 50, Green: 200, Blue: 10.It should be appreciated that the difference in pixel values based on the stream of image data representative of the interior 16 of the vehicle 10 is smaller than the difference in pixel values based on the stream of image data representative of the external environment 14 surrounding the vehicle 10. As mentioned above, this is because the stream of image data representative of the interior 16 of the vehicle 10 changes less than the stream of image data representative of the external environment 14 surrounding the vehicle 10.

[0041] In response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras 24 is less than or equal to the pixel value difference threshold, the frame differentiating module 52 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 may represent the interior 16 of the vehicle 10.In particular, if the frame analysis module 50 and the frame differentiation module 52 both determine that the image data acquired by the one or more peripheral cameras 24 potentially represents the interior of the vehicle 10, then the one or more controllers 20 determine that the image data acquired by the peripheral camera 24 represents the interior of the vehicle 10 and instruct the one or more vehicle systems 26 to discard the image data acquired by the peripheral camera 24. In one non-limiting embodiment, the frame differentiation module 52 may then send a signal 68 to the notification module 62 indicating that the image data acquired by the peripheral camera 24 represents the interior of the vehicle 10.The notification module 62 then instructs the one or more notification devices 28 to generate the notification to the occupant 40 indicating that the one or more peripheral cameras 24 are capturing image data of the interior environment representative of the interior 16 of the vehicle 10.

[0042] In response to either the frame differentiation module 52 determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras 24 is greater than the pixel value difference threshold, or the frame analysis module 50 determining that the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time is greater than the average delta frame size threshold, the one or more controllers 20 determine that the image data acquired by the one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10.The frame differentiation module 52 may then transmit the image data acquired by the one or more peripheral cameras 24 to the spatial motion vector analysis module 54 for spatial analysis.

[0043] The spatial motion vector analysis module 54 of the one or more controllers 20 determines a plurality of motion vectors 70 (in Fig. 3) that refer to a scene represented between two image frames that are part of the image data captured by the one or more peripheral cameras 24, based on one or more video compression algorithms. Some examples of video compression algorithms include, but are not limited to, AOMedia Video 1 (AV1) and Advanced Video Coding (AVC), also referred to as H.264. Fig. 3 is an exemplary illustration of one of the two image frames, wherein the image data captures a portion of a dashboard 80 of the vehicle 10 as well as the external environment 14 around the vehicle 10. Specifically, in Fig. 3 depicts the external environment 14 around the vehicle 10, vegetation 82 on the right side, which includes several trees, the sky 84 and a grassy area 86 on the left side.

[0044] Referring to Fig. 2 and Fig. 3, the spatial motion vector analysis module 54 of the one or more controllers 20 analyzes a magnitude of each of the plurality of motion vectors 70 between the two image frames to determine a change in motion corresponding to each motion vector 70. The spatial motion vector analysis module 54 of the one or more controllers 20 then maps one or more high-motion regions 90 and one or more low-motion regions 92 within the two image frames by comparing the change in motion corresponding to each motion vector 70 of the two image frames to a threshold motion value. The threshold motion value depends on the rate of motion of the vehicle 10, as indicated by the one or more vehicle dynamics parameters 64.

[0045] The threshold value for a motion value represents the change in motion detected between two image frames when the corresponding image data represents the interior 16 of the vehicle 10. If the change in motion of a motion vector 70 is greater than the threshold value for a motion value, then the motion vector 70 corresponds to image data representing the external environment 14 around the vehicle 10. In the example, as shown in Fig. 3, the vegetation 82 and the grass area 86 both represent high-motion areas 90, while the instrument panel 80 located within the interior 16 and the sky 84 both represent low-motion areas 92. The spatial motion vector analysis module 54 of the one or more controllers 20 determines a percentage of the two image frames that contain the high-motion areas 90 and compares the percentage of the two image frames that contain the high-motion areas 90 to a threshold percentage value. The threshold percentage value indicates that a non-trivial portion of the scene represented between the two image frames captures the external environment 14 around the vehicle 10. In one embodiment, the threshold percentage value is approximately twenty percent; however, other values may be used.

[0046] In response to determining that the percentage of the two image frames containing the high-motion regions 90 is at least equal to the threshold percentage value, the spatial motion vector analysis module 54 of the one or more controllers 20 determines that the image data acquired via the one or more peripheral cameras 24 may represent the external environment 14 surrounding the vehicle 10. Otherwise, the spatial motion vector analysis module 54 of the one or more controllers 20 determines that the image data acquired via the one or more peripheral cameras 24 may represent the interior 16 of the vehicle 10.

[0047] The spatial and temporal motion vector analysis module 56 of the one or more controllers 20 analyzes each macroblock of each image frame that is part of a plurality of image frames from the image data acquired by the one or more peripheral cameras 24 to determine areas of increased motion within the plurality of image frames, wherein the plurality of image frames are collected over a period of time. The spatial and temporal motion vector analysis module 56 of the one or more controllers 20 compares the size of a region representing the area of increased motion within the plurality of image frames to a coverage area threshold.In response to determining that the size of the area representing the area of increased motion is equal to or greater than the coverage area threshold, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 represents the external environment 14 around the vehicle 10.

[0048] The time period is determined based on the rate of movement of the vehicle 10, as indicated by the one or more vehicle dynamics parameters 64, a target confidence level, and the processing power of the one or more controllers 20. In one non-limiting embodiment, the time period is approximately ten seconds. The coverage area threshold is selected to exclude outlier data and disregard or discard trivial areas of the plurality of image frames containing sections of the external environment 14 that cover less than five percent of the total area of the image frame. In one non-limiting embodiment, the coverage area threshold is approximately twenty percent of the total area of the image.

[0049] Determining areas of increased motion within the plurality of image frames using the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 will now be described. The spatial and temporal motion vector analysis module 56 of the one or more controllers 20 first calculates a two-dimensional vector for each pixel that is part of each macroblock of each image frame that is part of the plurality of image frames. For example, in one embodiment, the macroblock is sized to include 256 pixels; however, it should be appreciated that the macroblocks may also contain a different number of pixels. The spatial and temporal motion vector analysis module 56 of the one or more controllers 20 then sums together all of the two-dimensional vectors corresponding to each pixel that is part of a particular macroblock.The spatial and temporal motion vector analysis module 56 of the one or more controllers 20 further adds all the two-dimensional vectors corresponding to each pixel that is part of a particular macroblock until the time period expires.

[0050] In response to determining that the time period has elapsed, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 determines a real length of a total motion vector corresponding to the particular macroblock based on the Pythagorean theorem formula. The spatial and temporal motion vector analysis module 56 of the one or more controllers 20 compares the real length of the total motion vector corresponding to the particular macroblock to a total motion vector threshold. The total motion vector threshold is selected to be representative of the motion observed in the external environment 14 around the vehicle 10.In response to determining that the real length of the overall motion vector corresponding to the particular macroblock is equal to or greater than the overall motion vector threshold, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 determines that the particular macroblock is part of an area of increased motion within the plurality of image frames.

[0051] It should be appreciated that in the described embodiment, the one or more vehicle systems 26 require image data of the external environment 14 to perform one or more tasks. Accordingly, in response to determining that the image data acquired by the one or more peripheral cameras 24 represents the external environment 14 surrounding the vehicle 10, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 transmits the image data acquired by the one or more peripheral cameras 24 to the one or more vehicle systems 26.Otherwise, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 represents the interior 16 of the vehicle 10 and instructs the one or more vehicle systems 26 to discard the image data. In one non-limiting embodiment, the spatial and temporal motion vector analysis module 56 transmits a signal 94 to the notification module 62 indicating that the image data acquired by the peripheral camera 24 represents the interior of the vehicle 10. The notification module 62 then instructs the one or more notification devices 28 to generate the notification to the occupant 40 indicating that the one or more peripheral cameras 24 are acquiring image data of the interior environment representative of the interior 16 of the vehicle 10.

[0052] In an alternative embodiment, the one or more vehicle systems 26 may require image data of the interior 16 of the vehicle 10, rather than the exterior environment 14, to perform one or more tasks. Accordingly, in response to determining that the image data acquired by the one or more peripheral cameras 24 is representative of the interior 16 of the vehicle 10, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 transmits the image data acquired by the one or more peripheral cameras 24 to the one or more vehicle systems 26.Otherwise, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 represents the external environment 14 and instructs the one or more vehicle systems 26 to discard the image data. In one embodiment, the spatial and temporal motion vector analysis module 56 may transmit the signal 94 to the notification module 62, and the notification module 62 then instructs the one or more notification devices 28 to generate the notification to the occupant 40 indicating that the one or more peripheral cameras 24 are acquiring image data of the external environment 14.

[0053] In one non-limiting embodiment, the edge detection module 58 of the one or more controllers 20 confirms that the image data acquired by the one or more peripheral cameras 24 is representative of the external environment 14 around the vehicle 10. In particular, the edge detection module 58 of the one or more controllers 20 executes one or more edge detection algorithms to identify lane markings located along a roadway in the image data acquired by the one or more peripheral cameras 24. In response to identifying lane markings within the image data acquired by the one or more peripheral cameras 24, the edge detection module 58 of the one or more controllers 20 confirms that the image data acquired by the one or more peripheral cameras 24 is representative of the external environment 14.It should be appreciated that confirming that the image data captured by the one or more peripheral cameras 24 is representative of the external environment 14 is optional and may be omitted in some embodiments.

[0054] In one embodiment, the re-evaluation module 60 of the one or more controllers 20 may continuously monitor an orientation angle θ and a position d of the one or more peripheral cameras 24 based on an ultra-wideband (UWB) sensor network 96. Referring to the two Fig. 1 and Fig. 2, the UWB sensor network 96 comprises three or more anchors 98 mounted on the vehicle 10 (in Fig. 1) are mounted, while a day 100 ( Fig. 1) of the UWB sensor network 96 is mounted on the one or more peripheral cameras 24. The tag 100 is a mobile sensor that is movable away from the vehicle 10 and transmits and receives sensor signals. Each anchor 98 of the UWB sensor network 96 is in wireless communication with the tag 100 to transmit and receive the sensor signals for tracking the tag 100.

[0055] The re-evaluation module 60 continuously monitors the UWB sensor network 96 with respect to the orientation angle θ and the position d of the one or more peripheral cameras 24 until it is determined that a change in the orientation angle θ, the position d, or both the orientation angle θ and the position d of the one or more cameras 24 has exceeded a threshold. In response to determining that the change in the orientation angle θ, the position d, or both the orientation angle θ and the position d of the one or more cameras 24 has exceeded the threshold, the re-evaluation module 60 instructs the one or more controllers 20 to re-evaluate whether the image data acquired by the one or more peripheral cameras 24 represents the interior 16 of the vehicle 10. The threshold indicates that the occupant 40 ( Fig. 1) Reposition the one or more peripheral cameras 24 to capture image data of a different environment. For example, although the one or more peripheral cameras 24 may have been positioned to capture image data of the external environment 14, the occupant 40 may choose to reposition the one or more peripheral cameras 24 to capture image data representative of the interior 16 instead.

[0056] Fig. 4 is a process flow diagram illustrating a method 400 for analyzing the data captured by the one or more peripheral cameras 24 by the Fig. 1 illustrated image analysis system 12 captured image data. Generally on Fig. 1, Fig. 2 and Fig. Referring to FIG. 4, the method 400 may begin with a decision block 402. At decision block 402, the one or more controllers 20 continue to monitor the one or more peripheral cameras 24 until the image data captured by the one or more peripheral cameras 24 is received. In response to receiving the image data, the method 400 proceeds to block 404.

[0057] At block 404, the frame analysis module 50 of the one or more controllers 20 classifies each image frame of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time as either a key frame or a delta frame. The method 400 may then proceed to block 406.

[0058] In block 406, the frame analysis module 50 of the one or more controllers 20 compares the average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time to the average delta frame size threshold. In response to determining that the average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time is less than or equal to the average delta frame size threshold, the frame analysis module 50 determines that the image data acquired by the one or more peripheral cameras 24 may represent the interior 16 of the vehicle 10.In response to determining that the average frame size of all the delta frames included in the image data acquired by the one or more peripheral cameras 24 over the predefined period of time is greater than the average delta frame size threshold, the frame analysis module 50 determines that the image data acquired by the one or more peripheral cameras 24 may represent the external environment 14 surrounding the vehicle 10. The method 400 may then proceed to block 408.

[0059] In block 408, the frame differentiation module 52 of the one or more controllers 20 determines the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras using the pixel value difference threshold, where the two image frames are spaced apart by the time interval. The method 400 may then proceed to a decision block 410.

[0060] At decision block 412, in response to the frame differentiation module 52 of the one or more controllers 20 determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras 24 is less than or equal to the threshold pixel value difference, and the frame differentiation module 52 of the one or more controllers 20 determining that the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras 24 over the predefined period of time is less than or equal to the average delta frame size threshold, the frame differentiation module 52 determines that the image data acquired by the one or more peripheral cameras represents the interior of the vehicle 10.The frame differentiation module 52 then instructs the one or more vehicle systems 26 to discard the image data acquired by the peripheral camera 24. The method 400 may then proceed to block 412.

[0061] At block 412, in response to determining that the image data captured by the one or more peripheral cameras 24 represents the interior 16 of the vehicle 10, the frame differentiation module 52 sends the signal 68 to the notification module 62 indicating that the image data captured by the peripheral camera 24 represents the interior of the vehicle 10. The notification module 62 then instructs the one or more notification devices 28 to generate the notification to the occupant 40 indicating that the one or more peripheral cameras 24 are capturing image data of the interior environment representative of the interior 16 of the vehicle 10. It should be appreciated that in embodiments, the notification to the occupant 40 is not necessarily sent. The method 400 may then end.

[0062] Returning to block 410, in response to the frame differentiation module 52 of the one or more controllers 20 determining that the difference in pixel values between two image frames that are part of the image data acquired via the one or more peripheral cameras 24 is greater than the pixel value difference threshold, or the frame differentiation module 52 of the one or more controllers 20 determining that the average frame size of all of the delta frames that are part of the image data acquired via the one or more peripheral cameras 24 over the predefined period of time is greater than the average delta frame size threshold, the frame differentiation module 52 determines that the image data acquired via the one or more peripheral cameras may represent the external environment 14 around the vehicle 10, and the method 400 continues to block 414.

[0063] In block 414, in response to determining that the image data acquired by the one or more peripheral cameras 24 may represent the external environment 14 around the vehicle 10, the spatial motion vector analysis module 54 of the one or more controllers 20 determines the plurality of motion vectors 70 (illustrated in Fig. 3) relating to a scene represented between two image frames that are part of the image data captured by the one or more peripheral cameras 24, based on one or more video compression algorithms. The method 400 may then proceed to block 416.

[0064] In block 416, the spatial motion vector analysis module 54 of the one or more controllers 20 analyzes the magnitude of each of the plurality of motion vectors 70 (illustrated in Fig.3) between the two image frames to determine a change in motion corresponding to each motion vector 70, and map one or more high-motion regions 90 and one or more low-motion regions 92 within the two image frames by comparing the change in motion corresponding to each motion vector 70 of the two image frames to a threshold motion value. The method 400 may then proceed to block 418.

[0065] In block 418, the spatial motion vector analysis module 54 of the one or more controllers 20 determines a percentage of the two image frames containing the high-motion regions 90 and compares the percentage of the two image frames containing the high-motion regions 90 to a threshold percentage value. In response to determining that the percentage of the two image frames containing the high-motion regions 90 is at least equal to the threshold percentage value, the spatial motion vector analysis module 54 of the one or more controllers 20 determines that the image data acquired via the one or more peripheral cameras 24 may represent the external environment 14 surrounding the vehicle 10.In response to determining that the percentage of the two image frames containing the high-motion regions 90 is less than the percentage threshold, the spatial motion vector analysis module 54 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 may represent the interior 16 of the vehicle 10. The method 400 may then proceed to block 420.

[0066] At block 420, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 analyzes each macroblock of each image frame that is part of the plurality of image frames from the image data acquired by the one or more peripheral cameras 24 to determine areas of increased motion within the plurality of image frames, the plurality of image frames being collected over a period of time. The method 400 may then proceed to decision block 422.

[0067] At decision block 422, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 compares the size of the region representing the area of increased motion within the plurality of image frames to the coverage area threshold. In response to determining that the image data acquired via the one or more peripheral cameras 24 represents the interior 16 of the vehicle 10, the spatial and temporal motion vector analysis module 56 then instructs the one or more vehicle systems 26 to discard the image data acquired via the peripheral camera 24. The method 400 may then proceed to block 424.

[0068] At block 424, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 transmits the signal 94 to the notification module 62 indicating that the image data captured by the peripheral camera 24 represents the interior of the vehicle 10. The notification module 62 then instructs the one or more notification devices 28 to generate the notification to the occupant 40 indicating that the one or more peripheral cameras 24 are capturing image data of the interior environment representative of the interior 16 of the vehicle 10. It should be understood that in embodiments, the notification to the occupant 40 is not necessarily sent. The method 400 may then end.

[0069] Returning to block 422, in response to determining that the size of the area representing the area of increased motion is equal to or greater than the coverage area threshold, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 determines that the image data acquired by the one or more peripheral cameras 24 represents the external environment 14 surrounding the vehicle 10. The method 400 may then proceed to block 426.

[0070] In block 426, in response to determining that the image data acquired by the one or more peripheral cameras 24 represents the external environment 14 around the vehicle 10, the spatial and temporal motion vector analysis module 56 of the one or more controllers 20 transmits the image data acquired by the one or more peripheral cameras 24 to one or more vehicle systems 26. The method 400 may then end.

[0071] Referring generally to the figures, the disclosed image analysis system provides various technical effects and advantages. In particular, the disclosed image analysis system provides a relatively lightweight approach for determining whether image data acquired by a peripheral camera is suitable for use by one or more vehicle systems, such as an autonomous driving system. It should be appreciated that the disclosed approach requires significantly fewer computational resources compared to various currently available computer vision-based approaches.

[0072] The controllers may refer to or be part of an electronic circuit, a combinational logic circuit, a field-programmable gate array (FPGA), a processor (shared, dedicated, or clustered) executing code, or a combination of some or all of the above elements. Furthermore, the controllers may be based on a microprocessor, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor may operate under the control of an operating system residing in memory. The operating system may manage computer resources such that computer program code embodied as one or more computer software applications, such as an application residing in memory, may include instructions executed by the processor.In an alternative embodiment, the processor may execute the application directly, in which case the operating system may be omitted.

[0073] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. Such variations should not be considered a departure from the spirit and scope of the present disclosure. legend

[0074] In the drawings, N stands for No and Y stands for Yes.

Claims

[1] An image analysis system for a vehicle that analyses image data captured by one or more peripheral cameras, the image analysis system comprising: one or more controllers, each of the one or more controllers including one or more processors that execute instructions to: to receive the image data captured by the one or more peripheral cameras; classify each image frame of the image data acquired by the one or more peripheral cameras over a predefined period of time as either a key frame or a delta frame; compare an average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time with a threshold value for the average delta frame size; determining a difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras with a threshold for the pixel value difference, wherein the two image frames are spaced apart by a time interval; in response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is less than or equal to the pixel value difference threshold, and determining that the average frame size of all the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is less than or equal to the average delta frame size threshold, determine that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle; and in response to determining that the image data acquired by the one or more peripheral cameras represents an interior of the vehicle, instruct one or more vehicle systems to discard the image data acquired by the peripheral camera. [2] The image analysis system of claim 1, wherein the one or more processors of the one or more controllers execute instructions to: in response to determining that the difference in pixel values between two image frames that are part of the image data acquired by the one or more peripheral cameras is greater than the pixel value difference threshold or the average frame size of all of the delta frames that are part of the image data acquired by the one or more peripheral cameras over the predefined period of time is greater than the average delta frame size threshold, determine that the image data acquired by the one or more peripheral cameras may represent an external environment around the vehicle. [3] The image analysis system of claim 2, wherein the one or more processors of the one or more controllers execute instructions to: in response to determining that the image data acquired by the one or more peripheral cameras may represent the external environment around the vehicle, determine, based on one or more video compression algorithms, a plurality of motion vectors related to a scene represented between two image frames that are part of the image data acquired by the one or more peripheral cameras. [4] The image analysis system of claim 3, wherein the one or more processors of the one or more controllers execute instructions to: analyze a magnitude of each of the plurality of motion vectors between the two image frames to determine a change in motion corresponding to each motion vector; and to map one or more high motion regions and one or more low motion regions within the two image frames by comparing the change in motion corresponding to each motion vector of the two image frames with a threshold motion value. [5] The image analysis system of claim 4, wherein the one or more processors of the one or more controllers execute instructions to: determine a percentage of the two image frames that contain the high-motion areas; compare the percentage of the two image frames containing the high-motion areas with a threshold percentage value; in response to determining that the percentage of the two image frames containing the high-motion regions is at least equal to the threshold percentage value, determine that the image data acquired by the one or more peripheral cameras may represent the external environment around the vehicle; in response to determining that the percentage of the two image frames containing the high motion regions is less than the percentage threshold, determine that the image data acquired by the one or more peripheral cameras may represent the interior of the vehicle. [6] The image analysis system of claim 5, wherein the one or more processors of the one or more controllers execute instructions to: analyze each macroblock of each image frame that is part of a plurality of image frames from the image data acquired by the one or more peripheral cameras to determine areas of increased motion within the plurality of image frames, wherein the plurality of image frames are collected over a period of time; compare a size of a region representing the area of increased motion within the plurality of image frames with a coverage area threshold; and in response to determining that the size of the area representing the area of increased motion is equal to or greater than the coverage area threshold, determine that the image data acquired by the one or more peripheral cameras represents the external environment around the vehicle. [7] The image analysis system of claim 6, wherein the one or more processors of the one or more controllers execute instructions to: in response to determining that the image data acquired by the one or more peripheral cameras represents the external environment around the vehicle, transmit the image data acquired by the one or more peripheral cameras to one or more vehicle systems. [8] The image analysis system of claim 6, wherein the one or more processors of the one or more controllers execute instructions to: in response to determining that the image data acquired by the one or more peripheral cameras represents the external environment around the vehicle, execute one or more edge detection algorithms to identify a lane marking disposed along a roadway; and in response to identifying lane markings in the image data acquired by the one or more peripheral cameras, to confirm that the image data is representative of the external environment around the vehicle. [9] The image analysis system of claim 1, wherein the one or more peripheral cameras are in electronic communication with the one or more controllers, and wherein the one or more peripheral cameras are part of a personal mobile device of a vehicle occupant. [10] The image analysis system of claim 1, wherein the one or more processors of the one or more controllers execute instructions to: in response to determining that the image data captured by the one or more peripheral cameras represents an interior of the vehicle, instruct one or more notification devices to generate a notification indicating that the one or more peripheral cameras capture image data of the interior of the vehicle.

Images