AI-adjusted roi encoding for improving license plate text clarity in recorded video

Abstract

An apparatus including an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to process the pixel data arranged as a video frame, detect a bounding box location for a vehicle in the video frame, perform license plate detection within the bounding box location to detect a license plate region, determine a first encoding parameter, determine a second encoding parameter for the license plate region, and generate an encoded video frame using the first encoding parameter outside the license plate region and the second encoding parameter within the license plate region. An offset can be applied to the first encoding parameter to determine the second encoding parameter. The second encoding parameter can provide legibility of text while maintaining an average bitrate of the encoded video the same as encoding a full one of the video frames using the first encoding parameter.

Description

Technical Field

[0001] The present invention relates generally to computer vision, and more specifically, to a method and / or apparatus for implementing AI-adjusted ROI encoding to improve the clarity of license plate text in recorded videos. Background Technology

[0002] Modern cameras are capable of capturing highly detailed images and videos. For example, 4K video can provide crisp and lifelike details. However, not all camera applications can utilize high-resolution cameras. High-resolution cameras are expensive, can lead to high processing resource consumption, high power consumption, and large file sizes. Some camera applications are bandwidth-limited. For example, a camera system transmitting encoded video over a wireless network (i.e., 4G / 5G / LTE) may have limited bandwidth available for transmitting captured video data compared to a hard-wired (i.e., Ethernet-connected) camera system. Encoding and / or compression are implemented to limit the bit rate of the output video. However, encoded video has a less subjective video quality, and text may become difficult to read.

[0003] Mobile cameras (e.g., vehicle-mounted cameras) present specific challenges when encoding at low bitrates. Moving scenes tend to change continuously, resulting in significant frame-to-frame changes in much of the visual content. For example, video encoded in H.265 for a constant bitrate of 2 Mbps to provide 1080p30 video can be used for road traffic scenes. Conventional video rate control algorithms are region-independent (i.e., they have no preference for video quality in any particular region). Low-bitrate video resulting from scenes with a large amount of moving content may have insufficient cell-specific bit allocation for clearly encoding the video, leading to poor text clarity.

[0004] Text clarity is important in video. Text provides crucial information about location (i.e., street signs, building names, addresses). Text clarity is particularly important for vehicle license plates. License plate characters should be recognizable in the output video. For example, license plate legibility is important for police and insurance investigations. Video with legible text is preferred and more convincing compared to text output already generated from the video. License plate readers are able to determine license plate numbers in text format. Optical character recognition can also extract text from images and videos. However, text is separate from images / videos and stored separately (or as metadata). Storing license plate numbers raises privacy concerns.

[0005] Higher bitrate video (i.e., twice or more) with higher resolution CMOS sensors (i.e., 4K image sensors) can record license plates with better text clarity. However, high-resolution image sensors are expensive, and the resulting file size is large.

[0006] The goal is to achieve AI-adjusted ROI encoding to improve the clarity of license plate text in recorded videos. Summary of the Invention

[0007] This invention relates to an apparatus including an interface and a processor. The interface can be configured to receive pixel data. The processor can be configured to: process pixel data arranged as video frames; perform computer vision operations on the uncompressed video frames; detect a bounding box location for a vehicle in response to the computer vision operations; perform license plate detection within the bounding box location to detect a region of interest (ROI) for the vehicle's license plate; determine first encoding parameters to generate an encoded video frame based on the uncompressed video frames; determine second encoding parameters for the ROI of the license plate; and generate an encoded video frame using the first encoding parameters outside the ROI and the second encoding parameters within the ROI. An offset can be applied to the first encoding parameters for the ROI to determine the second encoding parameters. The second encoding parameters can provide the sharpness of the license plate text within the ROI while maintaining the same average bitrate for the encoded video as if encoding a complete video frame using the first encoding parameters. Attached Figure Description

[0008] Embodiments of the invention will become apparent from the following detailed description and the appended claims and drawings.

[0009] Figure 1 This is an example diagram illustrating a camera according to an exemplary embodiment of the present invention that can implement AI-adjusted ROI encoding to improve the clarity of license plate text in recorded video.

[0010] Figure 2 This is a diagram showing an example edge device camera.

[0011] Figure 3 This is a diagram illustrating an example embodiment of the invention configured to provide a full-around view of a vehicle.

[0012] Figure 4 This is a block diagram showing the camera system.

[0013] Figure 5 This is a block diagram illustrating an AI-adjusted region of interest encoding pipeline.

[0014] Figure 6 This is a diagram illustrating the performance of computer vision operations on an example video frame to detect the location of a vehicle bounding box.

[0015] Figure 7 This is a diagram illustrating vehicle license plate detection at the block level of video frames.

[0016] Figure 8 This is a diagram showing example encoding parameter offsets applied to the region of interest in a license plate.

[0017] Figure 9 This is a diagram showing a portion of an encoded video frame with enhanced text clarity.

[0018] Figure 10 This is a diagram illustrating the performance of computer vision operations on an example video frame to detect the location of a road sign.

[0019] Figure 11 This is a diagram illustrating marker text detection at the block level of a video frame.

[0020] Figure 12 This is a flowchart illustrating a method for providing AI-adjusted ROI encoding to improve the clarity of license plate text in recorded videos.

[0021] Figure 13 This is a flowchart illustrating a method for tracking object positions so that ROI detection can be performed at frame intervals.

[0022] Figure 14 This is a flowchart illustrating a method for applying a positive offset to general coding parameters to provide a consistent bit rate.

[0023] Figure 15 This is a flowchart illustrating a method for filtering license plate locations based on distance. Detailed Implementation

[0024] Embodiments of the present invention include providing AI-adjusted ROI encoding to improve license plate text clarity in recorded videos, which may: (i) combine license plate detection with vehicle detection; (ii) implement one neural network to detect vehicles and another neural network to detect license plates; (iii) achieve text clarity in bandwidth-constrained videos; (iv) achieve text clarity while reducing file size for moving scenes; (v) filter text based on distance; (vi) apply different encoding parameters at the block level; (vii) reduce the video bitrate outside the license plate area; (viii) enhance text clarity for road signs; (ix) use object tracking to update the license plate position at intervals; and / or (x) be implemented as one or more integrated circuits.

[0025] Embodiments of the present invention can be configured to enhance text clarity in encoded video. Different encoding parameters can be selected for different regions of the video frame. Encoding parameters can be selected for specific regions of a coding block (e.g., at the macroblock level and / or at the coding tree block (CTB) level). Selecting different encoding parameters for different block regions in a video frame can maintain the video bitrate (e.g., maintain a constant output file size) while enhancing the readability of text in the encoded video frame.

[0026] Embodiments of the present invention can be configured to detect one or more vehicles in video frames (e.g., raw video frames, unencoded video frames, pre-processed video frames, downscaled video frames, uncompressed video frames, etc.). After a vehicle is detected, license plate detection can be performed within the vehicle's location. Performing vehicle detection can begin by limiting the search area for the license plate to a sub-part of the video frame. Limiting the search area allows for faster generation of detection results compared to searching the entire video frame and / or prevents false positives in license plate detection (e.g., license plates hanging on a wall, detecting other text that looks similar to a license plate, detecting discarded license plates, etc.). A block corresponding to the license plate's location can be selected as a region of interest (ROI). Multiple license plates can be detected in each video frame. Settings for encoding parameters can be selected based on the detected ROIs. Embodiments of the present invention can implement vehicle detection, license plate detection, and video recording based on regions of interest (ROIs).

[0027] In some embodiments, one or more neural networks may be implemented. The neural networks may be configured to perform vehicle detection and / or license plate detection. In one example, a first neural network may be trained and implemented to detect vehicles and / or determine the distance to vehicles. In another example, a second neural network may be trained and implemented to determine the location of a license plate within the vehicle's position and / or to determine a specific coded block corresponding to the license plate. The specific type of neural network implemented and / or the training data used to implement the neural network may vary depending on the design criteria of the specific implementation.

[0028] Embodiments of the present invention can be configured to provide a solution for situations where encoded video frames captured by an in-vehicle camera cannot provide sufficiently clear text of license plates on other moving vehicles. For example, the in-vehicle camera may be bandwidth-limited (e.g., streaming video over a wireless network such as 4G / 5G / LTE) and / or have a limited power budget, which may limit processing power and / or limit the video bitrate of the encoded video output. An AI-adjusted ROI encoding system can enable a relatively consistent average video bitrate (e.g., consistent between video frames where no license plate was detected and those where it was detected). The AI-adjusted ROI encoding system can be configured to detect the bounding box location of the license plate (e.g., a relatively small area of ​​the video frame) and apply ROI encoding. ROI encoding can be selected to provide clear text. For example, providing clear text may be a trade-off in providing subjective video quality. In some embodiments, a higher quantization parameter (e.g., QP) can be selected, which may provide poorer video quality in portions of the encoded video frame that may be outside the license plate's region of interest. Providing poorer video quality outside the license plate's region of interest can keep the video bitrate relatively constant. For example, AI-adjusted ROI encoding can achieve legible text using a lower-resolution CMOS image sensor (e.g., 1080p30 video) instead of relying on higher bitrate video (e.g., two times or more) from a higher-resolution CMOS sensor (e.g., 4K) used to record license plate text at higher definition. The resulting lower bitrate video can be achieved at a lower cost and / or lower power compared to a higher-resolution image sensor, and also saves network bandwidth (e.g., uploading the encoded video to a cloud service).

[0029] Embodiments of the present invention can combine vehicle detection with license plate detection. For example, a neural network can perform vehicle detection to first determine the location of a sub-region of a video frame that includes the vehicle, and then the neural network (e.g., a standalone neural network) can detect the license plate within the bounding box of the detected vehicle. Applying both vehicle detection and subsequent license plate detection can provide accuracy in license plate detection (e.g., preventing false positives) and increase the speed of license plate detection (e.g., compared to applying license plate detection across the entire video frame). Computer vision operations can be performed on raw and / or uncompressed video frames (e.g., YUV format video frames) to determine the bounding box location of the vehicle license plate. Encoding can then be applied to provide text clarity within the license plate bounding box by adjusting ROI encoding parameters (e.g., CTB for H.265 or macroblocks (MB) for H.264 encoding) while maintaining a consistent average video bitrate. Keeping the video bitrate relatively constant (e.g., regardless of whether the license plate is detected in the video frame) can save bits in video file storage. For example, maintaining a low average bitrate can save on storage capacity and / or network bandwidth costs.

[0030] Providing license plate detection to ensure the clarity of text within encoded video frames allows license plate characters to be legible within the encoded video frame. In one example, legible text can benefit end-users (e.g., police officers, insurance investigators, etc.) by enabling them to directly view and recognize the license plate text from the video output. Viewing legible text directly from the video output is likely easier and more convincing than relying on the output of a license plate reader system and / or an optical character recognition (OCR) system. For example, license plate readers and OCR can provide text output (e.g., as a separate data stream, a separate file, as metadata for the video, etc.), which may contain errors that cannot be verified if the source video is not available. Furthermore, directly storing the license plate text output raises privacy concerns (e.g., personally identifiable information may be subject to strict data protection protocols). For instance, scraping a database of OCR text results may be easier to perform than visually inspecting the video to read the actual text within the video content.

[0031] After performing vehicle (or vehicle) detection and detecting the license plate within the vehicle bounding box, ROI encoding parameters can be applied. The coded block corresponding to the license plate location (e.g., CTB for H.265 or MB for H.264) can be detected. For example, encoding parameters can be selected for the entire video frame, and an offset can be applied to a subset of the parameters used for encoding corresponding to the coded block location of the license plate. Negative offset values ​​can be applied to the QP within the ROI. Negative offset values ​​can reduce the QP, which allows for better encoding of the ROI. For example, the selected QP can be used to encode most of the encoded video frame, and only the offset QP value can be used to encode the smaller portion of the encoded video frame corresponding to the ROI.

[0032] Encoding parameters (e.g., QP values) can be parameters that can be changed in real time (e.g., instantaneously). For example, CTB can correspond to blocks of image / video frames, or MB can correspond to blocks of image / video frames. AI-adjusted ROI encoding can be configured to adjust block-level parameters in real time in response to the detection of license plate location. Real-time modification of encoding parameters can be particularly efficient for video data captured when the camera system is moving and other objects in the camera's field of view are moving (e.g., video content may be constantly changing from frame to frame due to significant movement).

[0033] In some embodiments, the license plate can be detected on each video frame. For example, detecting each license plate position in each video frame can provide the highest level of accuracy for license plate positioning. In some embodiments, the license plate position can be detected at predetermined intervals. For example, the license plate position can be detected every 2 video frames, every 4 video frames, or every fraction of a second. Typically, for efficient tracking, license plate positioning can be performed at relatively small frame intervals (e.g., if there are too many frames between detections, the tracking algorithm may not work due to large differences in object position unless the car is moving very slowly). Object tracking and / or predictive tracking of objects can be implemented to estimate the license plate position between license plate detection intervals. For example, estimating the license plate position can save computational resources compared to detecting the license plate position in each video frame (e.g., the neural network may not need to operate on each video frame). For example, tracking may include determining the speed and / or direction of the object's movement (e.g., the object's trajectory) to estimate the object's position in future video frames based on analysis of the object's movement on multiple previously captured video frames.

[0034] In some embodiments, AI-adjusted ROI encoding can maintain a stable average bit rate by increasing the QP for the portion of the video frame outside the ROI. For example, increasing the QP for most of the video frame (e.g., the region outside the ROI) can compensate for the reduced QP caused by using negative offsets within the license plate region. Typically, the license plate location may correspond to a relatively small portion of the video frame. For example, any increase in QP used to compensate for negative QP offsets within the license plate location may be relatively small. The video quality effect of increased QP outside the ROI may be imperceptible to the human eye.

[0035] The Region of Interest (ROI) can be the total block area of ​​a license plate. For example, an ROI can include several squares (e.g., coded blocks) that can be smaller than the bounding box of the license plate. The bounding box of the license plate can be a rectangle comprising multiple pixels (e.g., width * height). Each of the coded blocks can be a specific number of pixels to form a square. Each of the coded blocks can be a sub-part of the total number of pixels within the bounding box. Typically, most vehicles can have approximately the same size and / or have sizes within a similar size range. In some embodiments, the size of the vehicle's bounding box can be proportional to the distance of the vehicle from the image sensor.

[0036] In some embodiments, AI-adjusted ROI encoding can perform filtering on each license plate bounding box. Filtering can be configured to remove license plates determined to be too far from the image sensor (e.g., too far from the ego vehicle). For example, distant license plates may not have legible text in uncompressed video. Applying a QP value offset may not make unreadable text from uncompressed video legible. By ignoring license plates that may not result in legible text, computational resources can be saved. In some embodiments, the QP value offset can be adaptive to the distance to the detected license plate. For example, for each of the remaining (e.g., after filtering) license plates, a separate adaptive QP offset value can be determined. Individual adaptive QP offset values ​​can be selected based on the distance to the ego vehicle to ensure that each of the license plates is legible.

[0037] In some embodiments, AI-adjusted ROI encoding can be extended to other types of text. For example, the bounding box of a traffic sign can be used to enhance the clarity of text on the traffic sign. Other types of signs (e.g., text painted on roads, highway signs, building signs, etc.) can be detected to achieve text clarity in the encoded video frame. For example, the bounding box of a sign can be detected, the text location can be determined, and encoding parameters can be selected to provide clear text for the sign in the encoded video frame. The type of sign detected for AI-adjusted ROI encoding can vary depending on the design criteria of a particular implementation.

[0038] Typically, text clarity provided by embodiments of the present invention can include preventing the loss of data from uncompressed video frames. For example, an uncompressed video frame can include available video information (e.g., data including all information with optimal available clarity and / or image quality). Text clarity provided by embodiments of the present invention can reduce the amount of video data lost in uncompressed video frames during encoding. Data loss prevention can be limited to specific regions of interest (e.g., license plate data and / or other text determined to be of interest). In some embodiments, video processing can be performed to enhance text quality (e.g., using AI-based super-resolution operations). However, AI-based text enhancement can detect and redraw text based on probability. If the source video data (e.g., uncompressed video frames) does not display clear text, AI-based text enhancement may be a probabilistic game, potentially producing undesirable effects and / or visual artifacts and / or introducing errors.

[0039] If text is encoded in low-bitrate video, the higher loss in video encoding can generate blocky, blurry video and / or other artifacts, making the text in the video unrecognizable to the human eye or difficult for AI to detect and recognize. ROI encoding, implemented by embodiments of the present invention, can reduce this loss to generate an output encoded video frame with text that may look as good as the text in an uncompressed video frame. For example, without ROI encoding, the encoder can distribute bits evenly across the entire video frame. Since the entire video frame may have motion, determining which area has potentially more important data to achieve higher clarity can be difficult, potentially leading to a greater loss of encoded video quality in the license plate area compared to an uncompressed video frame.

[0040] refer to Figure 1 This diagram illustrates an example of a camera according to an exemplary embodiment of the invention, capable of implementing AI-adjusted ROI encoding to improve the clarity of license plate text in recorded video. A top view of zone 50 is shown. In the example shown, zone 50 may be an outdoor location. Streets, vehicles, and buildings are shown.

[0041] Devices 100a-100n are shown at various locations within zone 50. Devices 100a-100n can each implement an edge device. Edge devices 100a-100n may include intelligent IP cameras (e.g., camera systems). Edge devices 100a-100n may include low-power technologies designed for deployment in embedded platforms at the network edge (e.g., microprocessors running on sensors, cameras, or other battery-powered devices), where power consumption is a critical issue. In the examples, edge devices 100a-100n may include various traffic cameras and Intelligent Transportation System (ITS) solutions.

[0042] Edge devices 100a-100n can be implemented for a variety of applications. In the illustrated example, edge devices 100a-100n may include an Automatic License Plate Recognition (ANPR) camera 100a, a traffic camera 100b, a vehicle camera 100c, an access control camera 100d, an ATM camera 100e, a bullet camera 100f, a dome camera 100n, etc. In the example, edge devices 100a-100n can be implemented as a traffic camera and Intelligent Transportation System (ITS) solution designed to enhance road safety by utilizing a combination of people and vehicle detection, vehicle brand / model recognition, and Automatic License Plate Recognition (ANPR) functions.

[0043] In the illustrated example, zone 50 may be an outdoor location. In some embodiments, edge devices 100a-100n may be implemented at various indoor locations. In the example, edge devices 100a-100n may incorporate convolutional neural networks for use in security (surveillance) applications and / or access control applications. In the example, edge devices 100a-100n implemented as security cameras and access control applications may include battery-powered cameras, doorbell cameras, outdoor cameras, indoor cameras, etc. According to embodiments of the invention, security cameras and access control applications can realize performance benefits from the application of convolutional neural networks. In the example, edge devices utilizing convolutional neural networks according to embodiments of the invention can acquire large amounts of image data and perform on-device inference to obtain useful information (e.g., multiple temporal instances of images performed by each network), thereby reducing bandwidth and / or power consumption. In another example, security (surveillance) applications and / or location monitoring applications (e.g., tracking cameras) may benefit from large optical zoom. The design, type, and / or application performed by edge devices 100a-100n may vary depending on the design criteria of a particular implementation.

[0044] Camera systems 100a-100n can capture video in bandwidth-constrained environments within outdoor location area 50. For example, one or more of camera systems 100a-100n can be configured to connect to a wireless network (e.g., 4G / 5G / LTE), which may be bandwidth-constrained compared to a hardwired connection (e.g., Ethernet). In some embodiments, ATM camera 100e and / or access control camera 100d may be fixed cameras suitable for hardwired connections, but may also be configured to connect to a wireless connection (e.g., for ease of installation and / or cost savings). The environment in outdoor location area 50 can change in real time (e.g., capturing scenes that may include moving objects). Vehicle camera 100c may be particularly likely to capture moving scenes (e.g., vehicles may be moving, resulting in continuously changing scenes). Action cameras may also capture moving scenes. Scenes with high-speed moving objects may be susceptible to text that may be difficult to read after encoding. Each of camera systems 100a-100n can be configured to implement AI-adjusted ROI encoding.

[0045] refer to Figure 2 The diagram illustrates an example edge device camera. Camera systems 100a-100n are shown. Each camera device 100a-100n can have a different style and / or use case. For example, camera 100a can be an action camera, camera 100b can be a ceiling-mounted security camera, camera 100n can be a network camera, etc. Other types of cameras can be implemented (e.g., home security cameras, battery-powered cameras, doorbell cameras, stereo cameras, etc.). In some embodiments, camera systems 100a-100n can be fixed cameras (e.g., mounted and / or fixed in a single location). In some embodiments, camera systems 100a-100n can be handheld cameras. In some embodiments, camera systems 100a-100n can be configured for panning across areas, can be attached to mounts, gimbals, camera stands, etc. The design / style of cameras 100a-100n can vary depending on the design criteria of a particular implementation.

[0046] Each of the camera systems 100a-100n may include block (or circuit) 102, block (or circuit) 104, and / or block (or circuit) 106. Circuit 102 may implement a processor. Circuit 104 may implement a capture device. Circuit 106 may implement an inertial measurement unit (IMU). Camera systems 100a-100n may include other components (not shown). They can be used with... Figure 4 The components of the cameras 100a-100n are described in detail.

[0047] Processor 102 can be configured to implement an artificial neural network (ANN). In an example, the ANN may include a convolutional neural network (CNN). Processor 102 can be configured to implement a video encoder. Processor 102 can be configured to process pixel data arranged as video frames. Capture device 104 can be configured to capture pixel data that can be used by processor 102 to generate video frames. IMU 106 can be configured to generate motion data (e.g., vibration information, camera jitter, translation direction, etc.). In some embodiments, a structured light projector can be implemented to project a speckle pattern onto the environment. Capture device 104 can capture pixel data including a background image (e.g., the environment) with a speckle pattern. Although each of the cameras 100a-100n is shown as not implementing a structured light projector, some of the cameras 100a-100n may utilize a structured light projector (e.g., a camera implementing a sensor for capturing IR light).

[0048] Cameras 100a-100n can be edge devices. A processor 102 implemented by each of the cameras 100a-100n enables the cameras 100a-100n to perform various functions internally (e.g., at the local level). For example, processor 102 can be configured to perform object / event detection (e.g., computer vision operations), 3D reconstruction, presence detection, depth map generation, video encoding, electronic image stabilization, and / or video transcoding on the device. For example, processor 102 can even perform advanced processes such as computer vision and 3D reconstruction without uploading video data to a cloud service, thereby offloading computationally intensive functions (e.g., computer vision, video encoding, video transcoding, etc.).

[0049] In some embodiments, multiple camera systems may be implemented (e.g., camera systems 100a-100n may operate independently of each other). For example, each camera in 100a-100n may individually analyze the captured pixel data and perform event / object detection locally. In some embodiments, cameras 100a-100n may be configured as a camera network (e.g., security cameras that send video data to a central source such as a network-attached storage device and / or cloud service). The location and / or configuration of cameras 100a-100n may vary according to the design criteria of a particular implementation.

[0050] The capture device 104 of each camera in the camera systems 100a-100n may include a single lens (e.g., a monocular camera). The processor 102 may be configured to accelerate the preprocessing of speckle structured light for monocular 3D reconstruction. Monocular 3D reconstruction can be performed to generate depth maps and / or disparity maps without using a stereo camera.

[0051] refer to Figure 3 The diagram illustrates an example embodiment of the invention configured to provide a full-around view of a vehicle. An external environment 70 with vehicle 80 is shown. In the example shown, vehicle 80 may be a personal vehicle. In one example, vehicle 80 may be a commercial vehicle (e.g., a parcel delivery vehicle, service vehicle, public transport vehicle, etc.). In some embodiments, vehicle 80 may be a commercial truck (e.g., a semi-trailer truck). In some embodiments, vehicle 80 may be a pickup truck (e.g., a light vehicle, medium vehicle, heavy vehicle, etc.). In some embodiments, vehicle 80 may be a commuter vehicle and / or a family vehicle (e.g., a family vehicle such as a sedan, minivan, SUV, crossover, etc.). Vehicle 80 may be an internal combustion engine (ICE) vehicle, a diesel vehicle, a hybrid electric vehicle, a battery electric vehicle, etc. The type of vehicle 80 implemented may vary depending on the design criteria of a particular implementation.

[0052] External side mirrors 82a-82b are shown on vehicle 80. Side mirror 82a may be a driver-side side mirror of vehicle 80. Side mirror 82b may be a passenger-side side mirror of vehicle 80. Driver 90 is shown inside vehicle 80. Vehicle 80 may include devices 100a-100n. Devices 100a-100n may be camera systems. Camera systems 100a-100b are shown as integrated as part of vehicle 80. Camera system 100a is shown on the passenger side of vehicle 80. Camera system 100a is shown below passenger side mirror 82b. Camera system 100b is shown on the front grille of vehicle 80. In the view of the vehicle 80 shown, the three camera systems 100a-100b and 100e may be visible. However, one of the camera systems 100a-100n can be implemented horizontally below the driver's side mirror 82 (not visible from the perspective of the external view shown). Other camera systems 100a-100n can be positioned throughout the exterior and / or interior of the vehicle 80. The camera systems 100a-100n can be configured to capture a full-around view of the environment 70 surrounding the vehicle 80.

[0053] Dashed lines 92a-92e are shown. In the example shown, dashed line 92a is shown extending from camera system 100a, and dashed line 92b is shown extending from camera system 100b toward the exterior of the vehicle. Dashed lines 92c-92d can similarly extend from the corresponding camera systems 100c-100d (not visible from the shown viewpoint). Dashed lines 92a-92d can provide an illustrative representation of the field of view captured by each of the camera systems 100a-100d. Together, the fields of view 92a-92d can provide a full-around view of the environment surrounding vehicle 80.

[0054] All-around views 92a-92d are shown. In the example, all-around views 92a-92d can implement an all-around view (AVM) system. The AVM system may include four cameras (e.g., each camera may include one of camera systems 100a-100n and / or a combination of a pair of stereo lenses implemented by camera systems 100a-100n). In the viewpoint shown in environment 70, camera systems 100a and 100b may each be one of the four cameras, and the other two cameras may be invisible. In the example, camera system 100b may be a camera located on the front grille of vehicle 80, one of the cameras may be rearward (e.g., above the license plate), camera system 100a may be located below the passenger-side side mirror 82b, and one of the cameras may be located below the driver-side side mirror 82a. The arrangement of the cameras may vary depending on the design criteria of a particular implementation.

[0055] The dashed line 92e is shown extending from the camera system 100e toward the interior of the vehicle 80. The camera system 100e may be a passenger compartment surveillance camera system. The camera system 100e may be configured to capture a field of view 92e of the passenger compartment of the vehicle 80. The field of view 92e may be directed toward the driver 90. In some embodiments, the field of view 92e may be directed toward the driver 90 and / or other occupants of the vehicle 80.

[0056] In some embodiments, each of the camera systems 100a-100e may be configured to capture pixel data arranged as video frames. In some embodiments, each of the camera systems 100a-100d providing 360-degree views 92a-92d and / or the camera system 100e providing a cabin view may implement a fisheye lens (e.g., a 180-degree angular aperture may be used to capture video frames). The 360-degree views 92a-92d are shown as providing coverage of the field of view surrounding the vehicle 80. For example, a portion of 360-degree views 92a may provide coverage of the passenger side of the vehicle 80, a portion of 360-degree views 92b may provide coverage of the front of the vehicle 80, a portion of 360-degree views 92c may provide coverage of the driver side of the vehicle 80, and a portion of 360-degree views 92d may provide coverage of the rear of the vehicle 80. Each portion of the 360-degree views 92a-92d may be a field of view of a camera mounted to the vehicle 80. Each segment of the panoramic views 92a-92d can be de-distorted and stitched together by a video processor to provide enhanced video frames representing a top-down view of the vicinity of vehicle 80. Camera systems 100a-100d can be configured to implement a bird's-eye view transformer network (e.g., a deep learning model designed to generate BEV representations from multi-camera images). In this example, the panoramic views 92a-92d can be used to provide a bird's-eye view representation of vehicle 80.

[0057] Camera systems 100a-100e can provide representative examples of mechanisms for image acquisition. In one example, camera systems 100a-100e can be implemented as a monocular camera. In another example, camera systems 100a-100e can be implemented as stereo cameras (e.g., two capture devices implemented in a stereo pair). In some embodiments, the stereo cameras can be horizontally oriented. In some embodiments, the stereo cameras can be vertically oriented. In one example, four stereo cameras (e.g., eight capture devices) can be implemented, with one stereo camera on each side of vehicle 80. In some embodiments, camera systems 100a-100n can be installed as aftermarket products. For example, vehicle 80 can be sold without cameras, and one or more of camera systems 100n can be installed on vehicle 80. The implementation and / or location of camera systems 100a-100e on vehicle 80 and / or the orientation of camera systems 100a-100e can vary depending on the design criteria of a particular implementation.

[0058] Camera systems 100a-100d can capture scenes including continuous and / or near-continuous motion within the external environment 70. For example, vehicle 80 may be traveling through changing scenery, where objects may move relative to vehicle 80. Each of camera systems 100a-100e can be configured to implement AI-adjusted ROI encoding. Moving objects captured by camera systems 100a-100e may result in difficult-to-read text. AI-adjusted ROI encoding allows encoded video frames to store readable text while maintaining a relatively constant bitrate.

[0059] refer to Figure 4 The diagram illustrates a block diagram of a camera system. The camera system (or device) 100 can be... Figure 2 The cameras 100a-100n shown in association and / or with Figure 3 Representative examples of cameras 100a-100e are shown in association. Camera system 100 may include processor / SoC 102, capture device 104, and IMU 106.

[0060] The camera system 100 may further include blocks (or circuits) 150, 152, 154, 156, 158, 160, 164, and / or 166. Circuit 150 may implement a memory. Circuit 152 may implement a battery. Circuit 154 may implement a communication device. Circuit 156 may implement a wireless interface. Circuit 158 ​​may implement a general-purpose processor. Block 160 may implement an optical lens. Circuit 164 may implement one or more sensors. Circuit 166 may implement a human-machine interface (HID) device. In some embodiments, the camera system 100 may include a processor / SoC 102, a capture device 104, an IMU 106, a memory 150, a lens 160, a sensor 164, a battery 152, a communication module 154, a wireless interface 156, and a processor 158. In another example, camera system 100 may include a processor / SoC 102, a capture device 104, an IMU 106, a processor 158, a lens 160, and a sensor 164 as a single device, while memory 150, battery 152, communication module 154, and wireless interface 156 may be components of separate devices. Camera system 100 may include other components (not shown). The number, type, and / or arrangement of components in camera system 100 may vary depending on the design criteria of a particular implementation.

[0061] In some embodiments, processor 102 may be implemented as a video processor. In an example, processor 102 may be configured to receive three-sensor video input using a high-speed SLVS / MIPI-CSI / LVCMOS interface. In some embodiments, processor 102 may be configured to perform depth sensing in addition to generating video frames. In an example, depth sensing may be performed in response to depth information and / or vector light data captured in the video frames. In some embodiments, processor 102 may be implemented as a data stream vector processor. In an example, processor 102 may include a highly parallel architecture configured to perform image / video processing and / or radar signal processing.

[0062] Memory 150 can store data. Memory 150 can be implemented in various types of memory, including but not limited to cache, flash memory, memory card, random access memory (RAM), dynamic RAM (DRAM), etc. The type and / or size of memory 150 can vary according to the design criteria of a particular implementation. The data stored in memory 150 may correspond to video files, motion information (e.g., readings from sensor 164), video fusion parameters, image stabilization parameters, user input, computer vision models, feature sets, radar data cubes, radar detection and / or metadata information. In some embodiments, memory 150 can store reference images. Reference images can be used for computer vision operations, 3D reconstruction, automatic exposure, etc. In some embodiments, reference images may include reference structured light images.

[0063] Processor / SoC 102 can be configured to execute computer-readable code and / or procedural information. In various embodiments, the computer-readable code can be stored within processor / SoC 102 (e.g., microcode, etc.) and / or in memory 150. In one example, processor / SoC 102 can be configured to execute one or more artificial neural network models (e.g., face recognition CNN, object detection CNN, object classification CNN, 3D reconstruction CNN, presence detection CNN, etc.) stored in memory 150. In another example, memory 150 can store one or more directed acyclic graphs (DAGs) and one or more sets of weights or biases defining one or more artificial neural network models. In yet another example, memory 150 can store instructions for performing transformation operations (e.g., discrete cosine transform, discrete Fourier transform, fast Fourier transform, etc.). Processor / SoC 102 can be configured to receive input from memory 150 and / or present output to memory 150. Processor / SoC 102 can be configured to present and / or receive other signals (not shown). The number and / or type of inputs and / or outputs of the processor / SoC 102 can vary depending on the design criteria of a particular implementation. The processor / SoC 102 can be configured for low-power (e.g., battery) operation.

[0064] Battery 152 can be configured to store and / or supply power to components of camera system 100. The dynamic actuator mechanism for the rolling shutter sensor 130 can be configured to conserve power. Reduced power consumption allows camera system 100 to operate for extended periods using battery 152 without recharging. Battery 152 can be rechargeable. Battery 152 can be built-in (e.g., non-replaceable) or replaceable. Battery 152 can have an input for connection to an external power source (e.g., for charging). In some embodiments, device 100 can be powered by an external power source (e.g., battery 152 may not be implemented or may be implemented as a backup power source). Battery 152 can be implemented using various battery technologies and / or chemistry. The type of battery 152 implemented can vary depending on the design criteria of a particular implementation.

[0065] The communication module 154 can be configured to implement one or more communication protocols. For example, the communication module 154 and the wireless interface 156 can be configured to implement one or more of the following: IEEE 102.11, IEEE 102.15, IEEE 102.15.1, IEEE 102.15.2, IEEE 102.15.3, IEEE 102.15.4, IEEE 102.15.5, IEEE 102.20, and / or In some embodiments, the communication module 154 may be a hardwired data port (e.g., a USB port, a mini-USB port, a USB-C connector, an HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In some embodiments, the wireless interface 156 may also implement one or more protocols associated with a cellular communication network (e.g., GSM, CDMA, GPRS, UMTS, CDMA2000, 3GPP LTE, 4G / HSPA / WiMAX, SMS, etc.). In embodiments where the camera system 100 is implemented as a wireless camera, the protocols implemented by the communication module 154 and the wireless interface 156 may be wireless communication protocols. The type of communication protocol implemented by the communication module 154 may vary depending on the design criteria of the specific implementation.

[0066] Communication module 154 and / or wireless interface 156 can be configured to generate broadcast signals as output from camera system 100. The broadcast signals can transmit video data, parallax data, and / or (multiple) control signals to external devices. For example, the broadcast signals can be sent to cloud storage services (e.g., storage services capable of on-demand scaling). In some embodiments, communication module 154 may not transmit data until the processor / SoC 102 has performed video analysis and / or radar signal processing to determine that an object is in the field of view of camera system 100.

[0067] In some embodiments, the communication module 154 can be configured to generate a manual control signal. The manual control signal can be generated in response to a signal received from a user by the communication module 154. The manual control signal can be configured to activate the processor / SoC 102. The processor / SoC 102 can be activated in response to the manual control signal, regardless of the power state of the camera system 100.

[0068] In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to receive a feature set. The received feature set may be used to detect events and / or objects. For example, the feature set may be used to perform computer vision operations. The feature set information may include instructions for the processor 102 to determine which types of objects correspond to objects of interest and / or events.

[0069] In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to receive user input. User input allows the user to adjust operating parameters for various features implemented by the processor 102. In some embodiments, the communication module 154 and / or the wireless interface 156 may be configured to interact with an application (e.g., an app) (e.g., using an application programming interface (API)). For example, the application may be implemented on a smartphone to allow the end user to adjust various settings and / or parameters for various features implemented by the processor 102 (e.g., setting video resolution, selecting frame rate, selecting output format, setting tolerance parameters for 3D reconstruction, etc.).

[0070] Processor 158 can be implemented using general-purpose processor circuitry. Processor 158 may be operable to interact with video processing circuitry 102 and memory 150 to perform various processing tasks. Processor 158 may be configured to execute computer-readable instructions. In one example, the computer-readable instructions may be stored in memory 150. In some embodiments, the computer-readable instructions may include controller operations. Typically, input from sensor 164 and / or human-machine interface device 166 is shown to be received by processor 102. In some embodiments, general-purpose processor 158 may be configured to receive and / or analyze data from sensor 164 and / or HID 166 and make decisions in response to input. In some embodiments, processor 158 may send data to and / or receive data from other components of camera system 100, such as battery 152, communication module 154, and / or wireless interface 156. In some embodiments, processor 158 may implement an integrated digital signal processor (IDSP). For example, IDSP 158 may be configured to implement a warp engine. Which functions of the camera system 100 are executed by processor 102 and general-purpose processor 158 can vary depending on the design criteria of a particular implementation.

[0071] Lens 160 may be attached to capture device 104. Capture device 104 may be configured to receive an input signal (e.g., LIN) via lens 160. The signal LIN may be an optical input (e.g., an analog image). Lens 160 may be implemented as an optical lens. Lens 160 may provide zoom and / or focus features. In one example, capture device 104 and / or lens 160 may be implemented as a single lens assembly. In another example, lens 160 may be implemented separately from capture device 104.

[0072] Capture device 104 can be configured to convert input light LIN into computer-readable data. Capture device 104 can capture data received through lens 160 to generate raw pixel data. In some embodiments, capture device 104 can capture data received through lens 160 to generate a bitstream (e.g., generate video frames). For example, capture device 104 can receive focused light from lens 160. Lens 160 can be oriented, tilted, panned, scaled, and / or rotated to provide a target view from camera system 100 (e.g., a view of video frames, a view of panoramic video frames captured using multiple camera systems 100a-100n, a target image and reference image view for stereo vision, etc.). Capture device 104 can generate a signal (e.g., VIDEO). The signal VIDEO can be pixel data (e.g., a sequence of pixels that can be used to generate video frames). In some embodiments, the signal VIDEO can be video data (e.g., a sequence of video frames). The signal VIDEO can be presented to one of the inputs of processor 102. In some embodiments, the pixel data generated by the capture device 104 may be uncompressed and / or raw data generated in response to focused light from the lens 160. In some embodiments, the output of the capture device 104 may be a digital video signal.

[0073] In this example, capture device 104 may include block (or circuitry) 180, block (or circuitry) 182, and block (or circuitry) 184. Circuitry 180 may be an image sensor. Circuitry 182 may be a processor and / or logic unit. Circuitry 184 may be memory circuitry (e.g., a frame buffer). Lens 160 (e.g., a camera lens) may be oriented to provide a view of the environment surrounding camera system 100. Lens 160 may be designed to capture environmental data (e.g., light input LIN). Lens 160 may be a wide-angle lens and / or a fisheye lens (e.g., a lens capable of capturing a wide field of view). Lens 160 may be configured to capture and / or focus light for capture device 104. Typically, image sensor 180 is located behind lens 160. Based on the light captured from lens 160, capture device 104 may generate bitstream and / or video data (e.g., a signal VIDEO).

[0074] Capture device 104 can be configured to capture video image data (e.g., light collected and focused by lens 160). Capture device 104 can capture data received through lens 160 to generate a video bitstream (e.g., pixel data for a sequence of video frames). In various embodiments, lens 160 can be implemented as a fixed-focus lens. Fixed-focus lenses are generally advantageous for smaller size and lower power consumption. In examples, fixed-focus lenses can be used in battery-powered, doorbell, and other low-power camera applications. In some embodiments, lens 160 can be oriented, tilted, panned, zoomed, and / or rotated to capture the environment around camera system 100 (e.g., capturing data from the field of view). In examples, professional camera models can utilize active lens systems for enhanced functionality, remote control, etc.

[0075] The capture device 104 can convert received light into a digital data stream. In some embodiments, the capture device 104 can perform analog-to-digital conversion. For example, the image sensor 180 can perform photoelectric conversion on the light received by the lens 160. The processor / logic unit 182 can convert the digital data stream into a video data stream (or bitstream), a video file, and / or multiple video frames. In this example, the capture device 104 can present the video data as a digital video signal (e.g., VIDEO). The digital video signal can include video frames (e.g., continuous digital images and / or audio). In some embodiments, the capture device 104 can include a microphone for capturing audio. In some embodiments, the microphone can be implemented as a separate component (e.g., one of the sensors 164).

[0076] Video data captured by capture device 104 can be represented as a signal / bitstream / data VIDEO (e.g., a digital video signal). Capture device 104 can present the signal VIDEO to processor / SoC 102. The signal VIDEO can represent video frames / video data. The signal VIDEO can be a video stream captured by capture device 104. In some embodiments, the signal VIDEO may include pixel data operable by processor 102 (e.g., a video processing pipeline, image signal processor (ISP), etc.). Processor 102 can generate video frames in response to the pixel data in the signal VIDEO.

[0077] The signal VIDEO may include pixel data arranged as video frames. In some embodiments, the signal VIDEO may be an image including a background (e.g., the captured environment and / or objects) and a speckle pattern generated by a structured light projector. The signal VIDEO may include a single-channel source image. A single-channel source image may be generated in response to capturing pixel data using a monocular lens 160.

[0078] Image sensor 180 can receive input light LIN from lens 160 and convert the light LIN into digital data (e.g., a bitstream). For example, image sensor 180 can perform photoelectric conversion on light from lens 160. In some embodiments, image sensor 180 may have an additional margin that is not used as part of the image output. In some embodiments, image sensor 180 may not have an additional margin. In various embodiments, image sensor 180 can be implemented as an RGB sensor, RGB-IR sensor, RCCB sensor, monocular image sensor, stereo image sensor, thermal sensor, event-based sensor, etc. For example, image sensor 180 can be any type of sensor configured to provide sufficient output for a computer vision operation (e.g., neural network-based detection) to be performed on the output data. In some embodiments, image sensor 180 can be configured to generate an RGB-IR video signal. In a field of view illuminated only by infrared light, image sensor 180 can generate a monochrome (B / W) video signal. In a field of view illuminated by both IR and visible light, image sensor 180 can be configured to generate color information in addition to generating a monochrome video signal. In various embodiments, the image sensor 180 may be configured to generate video signals in response to visible light and / or infrared (IR) light.

[0079] In some embodiments, camera sensor 180 may include a rolling shutter sensor or a global shutter sensor. In an example, rolling shutter sensor 180 may implement an RGB-IR sensor. In some embodiments, capture device 104 may include a rolling shutter IR sensor and an RGB sensor (e.g., implemented as separate components). In an example, rolling shutter sensor 180 may be implemented as an RGB-IR rolling shutter complementary metal-oxide-semiconductor (CMOS) image sensor. In some embodiments, image sensor 190 may be implemented as a CMOS sensor configured to implement a Bayer mode. In one example, rolling shutter sensor 180 may be configured to assert a signal indicating the exposure time of the first line. In one example, rolling shutter sensor 180 may apply a mask to a monochrome sensor. In an example, the mask may include multiple cells containing a red pixel, a green pixel, a blue pixel, and an IR pixel. The IR pixel may contain red, green, and blue filter materials that efficiently absorb all light in the visible spectrum while allowing longer infrared wavelengths to pass through with minimal loss. In the case of a rolling shutter, as each row (or line) of the sensor begins to be exposed, all pixels in that row (or line) can begin to be exposed simultaneously.

[0080] Processor / logic unit 182 can convert the bitstream into human-readable content (e.g., video data that an average person can understand regardless of image quality, such as video frames and / or pixel data that can be converted into video frames by processor 102). For example, processor / logic unit 182 can receive raw (e.g., raw) data from image sensor 180 and generate (e.g., encode) video data (e.g., bitstream) based on the raw data. Capture device 104 may have memory 184 to store raw data and / or processed bitstreams. For example, capture device 104 may implement frame memory and / or buffer 184 to store (e.g., provide temporary storage and / or cache) one or more video frames (e.g., digital video signals). In some embodiments, processor / logic unit 182 can perform analysis and / or correction on the video frames stored in memory / buffer 184 of capture device 104. Processor / logic unit 182 can provide status information about the captured video frames.

[0081] IMU 106 can be configured to detect motion and / or movement of camera system 100. IMU 106 is shown as receiving signals (e.g., MTN). Signal MTN may include a combination of forces acting on camera system 100. Signal MTN may include movement, vibration, jitter, translation, abrupt changes, etc. Signal MTN may represent movement in three-dimensional space (e.g., movement in the X, Y, and Z directions). The type and / or amount of motion received by IMU 106 may vary depending on the design criteria of a particular implementation.

[0082] IMU 106 may include block (or circuitry) 186. Circuitry 186 may implement a motion sensor. In one example, motion sensor 186 may be a gyroscope. Gyroscope 186 may be configured to measure the amount of movement. For example, gyroscope 186 may be configured to detect the amount and / or direction of movement of signal MTN and convert that movement into electrical data. IMU 106 may be configured to determine the amount and / or direction of movement measured by gyroscope 186. IMU 106 may convert the electrical data from gyroscope 186 into a format readable by processor 102. IMU 106 may be configured to generate a signal (e.g., M_INFO). Signal M_INFO may include measurement information in a format readable by processor 102. IMU 106 may present signal M_INFO to processor 102. The number, type, and / or arrangement of components of IMU 106 and / or the number, type, and / or functionality of signals communicated by IMU 106 may vary according to design criteria for a particular implementation.

[0083] Sensor 164 can implement multiple sensors, including but not limited to motion sensors, ambient light sensors, proximity sensors (e.g., ultrasonic, radar, passive infrared, lidar, etc.), audio sensors (e.g., microphones), etc. In embodiments implementing a motion sensor, sensor 164 can be configured to detect motion anywhere (or some location outside the field of view) monitored by camera system 100. In various embodiments, motion detection can be used as a threshold for activating capture device 104. Sensor 164 can be implemented as an internal component of camera system 100 and / or an external component of camera system 100. In one example, sensor 164 can be implemented as a passive infrared (PIR) sensor. In another example, sensor 164 can be implemented as a smart motion sensor. In yet another example, sensor 164 can be implemented as a microphone. In embodiments implementing a smart motion sensor, sensor 164 may include a low-resolution image sensor configured to detect motion and / or people.

[0084] In various embodiments, sensor 164 may generate signals (e.g., SENS). The SENS may include various data (or information) collected by sensor 164. In an example, the SENS may include data collected in response to motion detected in a monitored field of view, ambient light levels in the monitored field of view, and / or sound picked up in the monitored field of view. However, other types of data may be collected and / or generated based on application-specific design criteria. The SENS may be presented to processor / SoC 102. In an example, sensor 164 may generate (assert) the SENS when motion is detected in a field of view monitored by camera system 100. In another example, sensor 164 may generate (assert) the SENS when audio is triggered in a field of view monitored by camera system 100. In yet another example, sensor 164 may be configured to provide directional information about motion and / or sound detected in the field of view. This directional information may also be communicated to processor / SoC 102 via the SENS.

[0085] HID 166 can implement an input device. For example, HID 166 can be configured to receive human input. In one example, HID 166 can be configured to receive password input from a user. In another example, HID 166 can be configured to receive user input to provide various parameters and / or settings to processor 102 and / or memory 150. In some embodiments, camera system 100 may include a keypad, touchpad (or screen), doorbell switch, and / or other human-machine interface device (HID) 166. In an example, sensor 164 can be configured to determine when an object approaches HID 166. In an example where camera system 100 is implemented as part of an access control application, capture device 104 can be activated to provide images for identifying people attempting access, and the lock zone and / or illumination for access touchpad 166 can be turned on. For example, a combination of input from HID 166 (e.g., a password or PIN code) can be combined with presence determination and / or depth analysis performed by processor 102 to achieve two-factor authentication. HID 166 can present a signal (e.g., USR) to processor 102. The signal USR may include input received by HID 166.

[0086] In an embodiment of the camera system 100 implementing a structured light projector, the structured light projector may include a structured light pattern lens and / or a structured light source. The structured light source may be configured to generate a structured light pattern signal (e.g., a speckle pattern) that can be projected onto an environment near the camera system 100. The structured light pattern may be captured by a capture device 104 as part of a light input LIN. The structured light pattern lens may be configured to emit structured light generated by the structured light source of the structured light projector while protecting the structured light source. The structured light pattern lens may be configured to decompose the laser pattern generated by the structured light source into a pattern array (e.g., a dense dot pattern array for a speckle pattern).

[0087] In the example, the structured light source can be implemented as an array of vertical-cavity surface-emitting lasers (VCSELs) and a lens. However, other types of structured light sources can be implemented to meet the design criteria of specific applications. In the example, the VCSEL array is generally configured to generate laser patterns (e.g., signal SLP). The lens is generally configured to decompose the laser pattern into an array of dense point patterns. In the example, the structured light source can be implemented as a near-infrared (NIR) source. In various embodiments, the light source of the structured light source can be configured to emit light at a wavelength of approximately 940 nanometers (nm), which is invisible to the human eye. However, other wavelengths can be utilized. In the example, wavelengths in the range of approximately 800 nm to 1000 nm can be utilized.

[0088] Processor / SoC 102 can receive signals VIDEO, M_INFO, SENS, and USR. Processor / SoC 102 can generate one or more video output signals (e.g., VIDOUT), one or more control signals (e.g., CTRL), one or more depth data signals (e.g., DIMAGES), and / or one or more twist table data signals (e.g., WT) based on signals VIDEO, M_INFO, SENS, USR, and / or other inputs. In some embodiments, signals VIDOUT, DIMAGES, WT, and CTRL can be generated based on analysis of signals VIDEO and / or objects detected in signals VIDEO. In some embodiments, signals VIDOUT, DIMAGES, WT, and CTRL can be generated based on analysis of signals VIDEO, motion information captured by IMU 106, and / or inherent properties of lens 160 and / or capture device 104.

[0089] In various embodiments, the processor / SoC 102 may be configured to perform one or more of the following: feature extraction, object detection, object tracking, electronic image stabilization, 3D reconstruction, presence detection, and object identification. For example, the processor / SoC 102 may determine motion information and / or depth information by analyzing frames from a signal VIDEO and comparing those frames with previous frames. The comparison may be used to perform digital motion estimation. In some embodiments, the processor / SoC 102 may be configured to generate a video output signal VIDOUT, a distortion table data signal WT, and / or a depth data signal DIMAGES, including a disparity map and a depth map, based on the signal VIDEO. The video output signal VIDOUT, the distortion table data signal WT, and / or the depth data signal DIMAGES may be presented to memory 150, communication module 154, and / or wireless interface 156. In some embodiments, the video signal VIDOUT, the distortion table data signal WT, and / or the depth data signal DIMAGES may be used internally by the processor 102 (e.g., not presented as output). In one example, the warped table data signal WT can be used by a warping engine implemented by a digital signal processor (e.g., processor 158).

[0090] The signal VIDOUT may be presented to the communication module 154 and / or the wireless interface 156. In some embodiments, the signal VIDOUT may include encoded video frames generated by the processor 102. In some embodiments, the encoded video frames may include a complete video stream (e.g., encoded video frames representing all video captured by the capture device 104). The encoded video frames may be encoded, cropped, stitched, stabilized, and / or enhanced versions of pixel data received from the signal VIDEO. In an example, the encoded video frames may be high-resolution, digital, encoded, de-distorted, stabilized, cropped, blended, stitched, and / or rolled shutter effect corrected versions of the signal VIDEO.

[0091] In some embodiments, the signal VIDOUT can be generated based on video analysis (e.g., computer vision operations) performed by processor 102 on generated video frames. Processor 102 can be configured to perform computer vision operations to detect objects and / or events in the video frames, and then convert the detected objects and / or events into statistics and / or parameters. In one example, the data determined by the computer vision operations can be converted by processor 102 into a human-readable format. The data from the computer vision operations can be used to detect objects and / or events. The computer vision operations can be performed locally by processor 102 (e.g., without needing to communicate with external devices to offload computational operations). Similarly, other video processing and / or encoding operations (e.g., stabilization, compression, stitching, cropping, rolling shutter effect correction, etc.) can be performed locally by processor 102. For example, locally executed computer vision operations allow the computer vision operations to be performed by processor 102 and avoid heavy video processing running on a backend server. Avoiding video processing running on a backend (e.g., remotely located) server can protect privacy.

[0092] In some embodiments, the signal VIDOUT can be data generated by processor 102 (e.g., video analysis results, audio / speech analysis results, stabilized video frames, etc.) that can be communicated to cloud computing services to aggregate information and / or provide training data for machine learning (e.g., to improve object detection, improve audio detection, improve presence detection, etc.). In some embodiments, the signal VIDOUT can be provided to cloud services for mass storage (e.g., to enable users to retrieve encoded video using smartphones and / or desktop computers). In some embodiments, the signal VIDOUT can include data extracted from video frames (e.g., computer vision results) and can communicate the results to another device (e.g., a remote server, cloud computing system, etc.) to offload the analysis of the results to another device (e.g., offloading the analysis of the results to a cloud computing service instead of performing all analysis locally). The type of information communicated by the signal VIDOUT can vary depending on the design criteria of a particular implementation.

[0093] The CTRL signal can be configured to provide a control signal. The CTRL signal can be generated in response to a decision made by the processor 102. In one example, the CTRL signal can be generated in response to a detected object and / or features extracted from a video frame. The CTRL signal can be configured to enable, disable, or change the operating mode of another device. In one example, the CTRL signal can be used to lock / unlock a door controlled by an electronic lock. In another example, the CTRL signal can be used to put a device into sleep mode (e.g., low power mode) and / or activate it from sleep mode. In yet another example, the CTRL signal can be used to generate an alarm and / or notification. The type of device controlled by the CTRL signal and / or the response performed by the device in response to the CTRL signal can vary depending on the design criteria of the specific implementation.

[0094] A signal CTRL can be generated based on data received by sensor 164 (e.g., temperature readings, motion sensor readings, etc.). A signal CTRL can be generated based on input from HID 166. A signal CTRL can be generated based on human behavior detected by processor 102 in a video frame. A signal CTRL can be generated based on the type of object detected (e.g., person, animal, vehicle, etc.). A signal CTRL can be generated in response to detecting a specific type of object at a specific location. A signal CTRL can be generated in response to user input to provide various parameters and / or settings to processor 102 and / or memory 150. Processor 102 can be configured to generate a signal CTRL in response to sensor fusion operations (e.g., aggregating information received from different sources). Processor 102 can be configured to generate a signal CTRL in response to the result of an presence detection performed by processor 102. The conditions used to generate the signal CTRL can vary depending on the design criteria of a particular implementation.

[0095] Signals DIMAGES may include one or more depth maps and / or disparity maps generated by processor 102. Signals DIMAGES may be generated in response to 3D reconstruction performed on a monocular single-channel image. Signals DIMAGES may be generated in response to analysis of captured video data and structured light patterns.

[0096] A multi-step approach can be implemented to activate and / or disable the capture device 104 based on the output of the motion sensor 164 and / or any other power consumption characteristics of the camera system 100, thereby reducing the power consumption of the camera system 100 and extending the lifespan of the battery 152. The motion sensor in sensor 164 may have low power consumption on the battery 152 (e.g., less than 10W). In the example, the motion sensor in sensor 164 may be configured to remain on (e.g., always active) unless disabled in response to feedback from the processor / SoC 102. Video analysis performed by the processor / SoC 102 may have relatively high power consumption on the battery 152 (e.g., greater than that of the motion sensor 164). In the example, the processor / SoC 102 may be in a low-power state (or powered down) until some motion is detected by the motion sensor in sensor 164.

[0097] Camera system 100 can be configured to operate using various power states. For example, in a power-down state (e.g., sleep state, low power state), the motion sensor in sensor 164 and processor / SoC 102 can be turned on, and other components of camera system 100 (e.g., image capture device 106, memory 150, communication module 154, etc.) can be turned off. In another example, camera system 100 can operate in an intermediate state. In the intermediate state, image capture device 106 can be turned on, and memory 150 and / or communication module 154 can be turned off. In yet another example, camera system 100 can operate in a power-on (or high-power) state. In the power-on state, sensor 164, processor / SoC 102, capture device 104, memory 150, and / or communication module 154 can be turned on. Camera system 100 can consume some power from battery 152 (e.g., a relatively small and / or minimal amount of power) in the power-down state. In the power-on state, camera system 100 can consume more power from battery 152. The number of components of the camera system 100 that are turned on when the camera system 100 is in power state and / or operating in each power state can vary according to the design criteria of a particular implementation.

[0098] In some embodiments, the camera system 100 may be implemented as a system-on-a-chip (SoC). For example, the camera system 100 may be implemented as a printed circuit board including one or more components. The camera system 100 may be configured to perform intelligent video analysis on video frames of a video. The camera system 100 may be configured to crop and / or enhance the video.

[0099] In some embodiments, the video frame may be a view (or a derivative of a view) captured by the capture device 104. Pixel data signals may be enhanced by the processor 102 (e.g., color conversion, noise filtering, automatic exposure, automatic white balance, automatic focus, etc.). In some embodiments, the video frame may provide a series of cropped and / or enhanced video frames that improve the view from the perspective of the camera system 100 (e.g., providing night vision, providing high dynamic range (HDR) imaging, providing more viewing area, highlighting detected objects, providing additional data (e.g., digital distance to the detected object), etc.) to enable the processor 102 to see the location better than a human can see with human vision.

[0100] Encoded video frames can be processed locally. In one example, the encoded video can be stored locally by memory 150, enabling processor 102 to facilitate computer vision analysis internally (e.g., without first uploading the video frames to a cloud service). Processor 102 can be configured to select video frames to be encapsulated into a video stream that can be transmitted over a network (e.g., a bandwidth-limited network).

[0101] In some embodiments, processor 102 may be configured to perform sensor fusion operations. The sensor fusion operations performed by processor 102 may be configured to analyze information from multiple sources (e.g., capture device 104, IMU 106, sensor 164, and HID 166). By analyzing various data from different sources, sensor fusion operations may be able to make inferences about the data that might not be possible from just one data source. For example, sensor fusion operations implemented by processor 102 may analyze video data (e.g., mouth movements) as well as speech patterns from directional audio. Different sources can be used to develop scene models to support decision-making. For example, processor 102 may be configured to compare the synchronization of detected speech patterns with mouth movements in video frames to determine which person is speaking in the video frame. Sensor fusion operations may also provide temporal correlation, spatial correlation, and / or reliability between the received data.

[0102] In some embodiments, processor 102 may implement convolutional neural network (CNN) capabilities. CNN capabilities can be implemented using deep learning techniques for computer vision. CNN capabilities can be configured to perform pattern and / or image recognition using a training process involving multi-layer feature detection. Computer vision and / or CNN capabilities can be executed locally by processor 102. In some embodiments, processor 102 may receive training data and / or feature set information from external sources. For example, external devices (e.g., cloud services) may access various data sources to provide training data that camera system 100 may not be able to obtain. However, computer vision operations performed using feature sets can be performed using the computational resources of processor 102 within camera system 100.

[0103] The video pipeline of processor 102 can be configured to perform local dewarping, cropping, enhancement, rolling shutter correction, stabilization, downsizing, encapsulation, compression, conversion, mixing, synchronization, and / or other video operations. The video pipeline of processor 102 can support multiple streams (e.g., generating multiple bitstreams in parallel, each including a different bitrate). In the example, the video pipeline of processor 102 can implement an image signal processor (ISP) with an input pixel rate of 320 Mbps. The architecture of the video pipeline of processor 102 enables real-time and / or near-real-time video operations on high-resolution video and / or high-bitrate video data. The video pipeline of processor 102 can perform computer vision processing, stereo vision processing, object detection, 3D noise reduction, fisheye lens correction (e.g., real-time 360-degree dewarping and lens distortion correction), oversampling, and / or high dynamic range processing on 4K resolution video data. In one example, the architecture of the video pipeline can achieve 4K ultra-high resolution with H.264 encoding at twice the real-time speed (e.g., 60fps) and 4K ultra-high resolution with H.265 / HEVC and / or 4K AVC encoding at 30fps (e.g., 4KP30 AVC and HEVC encoding with multi-stream support). The type of video operation and / or the type of video data operated on by the processor 102 can vary according to the design criteria of the specific implementation.

[0104] In this embodiment, the camera sensor 180 can be implemented as a high-resolution sensor. Using the high-resolution sensor 180, the processor 102 can combine oversampling of the image sensor 180 with digital scaling within the cropped area. Oversampling and digital scaling can each be one of the video operations performed by the processor 102. Oversampling and digital scaling can be implemented to provide a higher resolution image within the total size limit of the cropped area. In some embodiments, the camera sensor 180 can be implemented as a low-cost CMOS sensor. For example, the CMOS sensor 180 can be configured to capture 1080p resolution video.

[0105] In some embodiments, lens 160 may be a fisheye lens. One of the video operations implemented by processor 102 may be a dedistortion operation. Processor 102 may be configured to dedistort generated video frames. Dedistortion may be configured to reduce and / or remove severe distortion caused by fisheye lens and / or other lens characteristics. For example, dedistortion may reduce and / or eliminate bulging effects to provide a linear image.

[0106] Processor 102 can be configured to crop (e.g., trim) a region of interest from a full video frame (e.g., generate a region of interest video frame). Processor 102 can generate video frames and select regions. In the example, cropping the region of interest can generate a second image. The cropped image (e.g., the region of interest video frame) can be smaller than the original video frame (e.g., the cropped image can be a portion of the captured video).

[0107] The region of interest (ROI) can be dynamically adjusted based on the location of the audio source. For example, the detected audio source may be moving, and its location may shift as video frames are captured. Processor 102 can update the coordinates of the selected ROI and dynamically update the cropped portion (e.g., a directional microphone implemented as one or more sensors in sensor 164 can dynamically update its location based on captured directional audio). The cropped portion may correspond to the selected ROI. As the ROI changes, the cropped portion may change. For example, the selected coordinates of the ROI may change from frame to frame, and processor 102 may be configured to crop the selected region in each frame.

[0108] Processor 102 can be configured to oversample image sensor 180. Oversampling of image sensor 180 can produce a higher resolution image. Processor 102 can be configured to digitally upscale regions of video frames. For example, processor 102 can digitally upscale a cropped region of interest. For example, processor 102 can establish a region of interest based on directional audio, crop the region of interest, and then digitally upscale the cropped region of interest video frame.

[0109] The dedistortion operation performed by processor 102 can adjust the visual content of the video data. The adjustment performed by processor 102 can make the visual content appear natural (e.g., as if seen by a person viewing a position corresponding to the field of view of capture device 104). In the example, dedistortion can alter the video data to generate linear video frames (e.g., correcting artifacts caused by lens characteristics of lens 160). Dedistortion operations can be implemented to correct distortion caused by lens 160. Adjusted visual content can be generated to achieve more accurate and / or reliable object detection.

[0110] Various features (e.g., de-distortion, digital scaling, cropping, etc.) can be implemented as hardware modules in processor 102. Implementing hardware modules can increase the video processing speed of processor 102 (e.g., faster than software implementations). Hardware implementations enable video processing while reducing latency. The hardware components used can vary depending on the design standards of the specific implementation.

[0111] In some embodiments, processor 102 may implement one or more coprocessors, cores, and / or chiplets. For example, processor 102 may implement one coprocessor configured as a general-purpose processor and another coprocessor configured as a video processor. In some embodiments, processor 102 may be a dedicated hardware module designed to perform a specific task. In one example, processor 102 may implement an AI accelerator. In another example, processor 102 may implement a radar processor. In yet another example, processor 102 may implement a dataflow vector processor. In some embodiments, other processors implemented by device 100 may be general-purpose processors and / or video processors (e.g., coprocessors that are physically different from processor 102 in chipset and / or silicon). In one example, processor 102 may implement the x86-64 instruction set. In another example, processor 102 may implement the ARM instruction set. In yet another example, processor 102 may implement the RISC-V instruction set. The number of cores, coprocessors, design optimizations, and / or instruction sets implemented by processor 102 may vary depending on the design criteria of a particular implementation.

[0112] The processor 102 shown includes multiple blocks (or circuits) 190a-190n. Blocks 190a-190n can implement various hardware modules implemented by the processor 102. Hardware modules 190a-190n can be configured to provide various hardware components to implement video processing pipelines, radar signal processing pipelines, and / or AI processing pipelines. Circuits 190a-190n can be configured to receive pixel data (VIDEO), generate video frames based on the pixel data, perform various operations on the video frames (e.g., de-distortion, rolling shutter correction, cropping, upscaling, image stabilization, 3D reconstruction, presence detection, automatic exposure, etc.), prepare video frames for communication with external hardware (e.g., encoding, encapsulation, color correction, etc.), parse feature sets, and implement various computer vision operations (e.g., object detection, segmentation, classification, etc.). Hardware modules 190a-190n can be configured to implement various security features (e.g., secure boot, I / O virtualization, etc.). Various implementations of processor 102 may not necessarily utilize all the features of hardware modules 190a-190n. The features and / or functions of hardware modules 190a-190n may vary depending on the design criteria of a particular implementation. Details of hardware modules 190a-190n may be described in connection with U.S. Patent Application No. 16 / 831,549, filed April 16, 2020; U.S. Patent Application No. 16 / 288,922, filed February 28, 2019; U.S. Patent Application No. 15 / 593,493 (now U.S. Patent No. 10,437,600), filed May 12, 2017; U.S. Patent Application No. 15 / 931,942, filed May 14, 2020; U.S. Patent Application No. 16 / 991,344, filed August 12, 2020; and U.S. Patent Application No. 17 / 479,034, filed September 20, 2021, the appropriate portions of which are incorporated herein by reference in their entirety.

[0113] Hardware modules 190a-190n can be implemented as dedicated hardware modules. Compared to software implementation, using dedicated hardware modules 190a-190n to implement various functions of processor 102 allows processor 102 to be highly optimized and / or customized to limit power consumption, reduce heat generation, and / or increase processing speed. Hardware modules 190a-190n can be customizable and / or programmable to implement multiple types of operations. Implementing dedicated hardware modules 190a-190n allows the hardware used to perform each type of computation to be optimized for speed and / or efficiency. For example, hardware modules 190a-190n can implement several relatively simple operations frequently used in computer vision operations, which together enable computer vision operations to be performed in real time. The video pipeline can be configured to identify objects. Objects can be identified by interpreting numerical and / or symbolic information to determine the specific type and / or characteristics of the object represented by the visual data. For example, the number of pixels and / or the color of pixels in video data can be used to identify portions of video data as objects. Hardware modules 190a-190n enable computationally intensive operations (e.g., computer vision operations, video encoding, video transcoding, 3D reconstruction, depth map generation, presence detection, etc.) to be performed locally by the camera system 100.

[0114] One of the hardware modules 190a-190n (e.g., 190a) can implement a scheduler circuit. Scheduler circuit 190a can be configured to store a directed acyclic graph (DAG). In the example, scheduler circuit 190a can be configured to generate and store a DAG in response to received (e.g., loaded) feature set information. The DAG can define video operations to be performed to extract data from video frames. For example, the DAG can define various mathematical weights (e.g., neural network weights and / or biases) to be applied when performing computer vision operations to classify various groups of pixels into specific objects.

[0115] Scheduler circuit 190a can be configured to parse acyclic graphs to generate various operators. Operators can be scheduled by scheduler circuit 190a in one or more of the other hardware modules 190a-190n. For example, one or more hardware modules 190a-190n can implement a hardware engine configured to perform a specific task (e.g., a hardware engine designed to perform repetitive specific mathematical operations used for performing computer vision operations). Scheduler circuit 190a can schedule operators based on when they are ready to be processed by hardware engines 190a-190n.

[0116] Scheduler circuit 190a can time-multiplex tasks to hardware modules 190a-190n for execution based on the availability of hardware modules 190a-190n. Scheduler circuit 190a can parse a directed acyclic graph into one or more data streams. Each data stream can include one or more operators. Once the directed acyclic graph is parsed, scheduler circuit 190a can allocate data streams / operators to hardware engines 190a-190n and send relevant operator configuration information to initiate the operators.

[0117] The binary representation of each directed acyclic graph can be an ordered traversal of the directed acyclic graph, where descriptors and operators are interleaved based on data dependencies. Descriptors typically provide registers that link data buffers to specific operands in a dependent operator. In various embodiments, operators may not appear in the directed acyclic graph representation until all dependent descriptors have been declared for operands.

[0118] One of the hardware modules 190a-190n (e.g., 190b) can implement an Artificial Neural Network (ANN) module. The ANN module can be implemented as a fully connected neural network or a Convolutional Neural Network (CNN). In this example, the fully connected network is "structure-agnostic" because no special assumptions need to be made about the input. A fully connected neural network consists of a series of fully connected layers that connect each neuron in one layer to each neuron in another layer. In a fully connected layer, for n inputs and m outputs, there are n*m ​​weights. There are also bias values ​​for each output node, resulting in a total of (n+1)*m parameters. In a trained neural network, (n+1)*m parameters have been determined during the training process. A trained neural network typically includes an architectural specification and a set of parameters (weights and biases) determined during training. In another example, a CNN architecture may explicitly assume that the input is an image, enabling the encoding of specific attributes into the model architecture. A CNN architecture may include a sequence of layers, where each layer transforms one activation into another through a differentiable function.

[0119] In the example shown, the artificial neural network 190b can implement a convolutional neural network (CNN) module. The CNN module 190b can be configured to perform computer vision operations on video frames. The CNN module 190b can be configured to achieve object recognition through multi-layer feature detection. The CNN module 190b can be configured to compute descriptors based on the performed feature detections. The descriptors enable the processor 102 to determine the probability that a pixel in a video frame corresponds to a specific object (e.g., a specific manufacture / model / year of a vehicle, identifying a person as a specific individual, detecting the type of animal, detecting facial features, etc.).

[0120] CNN module 190b can be configured to implement convolutional neural network capabilities. CNN module 190b can be configured to use deep learning techniques to implement computer vision. CNN module 190b can be configured to use a training process through multi-layer feature detection to implement pattern and / or image recognition. CNN module 190b can be configured to perform inference against machine learning models.

[0121] The CNN module 190b can be configured to perform feature extraction and / or matching solely in hardware. Feature points typically represent regions of interest (e.g., corners, edges, etc.) in a video frame. By temporarily tracking feature points, estimates of the self-motion of the capture platform or motion models of objects observed in a scene can be generated. To track feature points, a matching operation is typically incorporated into the CNN module 190b by hardware to find the most probable correspondence between feature points in a reference video frame and a target video frame. During the matching of reference and target feature points, each feature point can be represented by a descriptor (e.g., image patch, SIFT, BRIEF, ORB, FREAK, etc.). Implementing the CNN module 190b using dedicated hardware circuitry allows for real-time computation of descriptor matching distances.

[0122] The CNN module 190b can be configured to perform face detection, face recognition, and / or presence determination. For example, face detection, face recognition, and / or presence determination can be performed based on a trained neural network implemented by the CNN module 190b. In some embodiments, the CNN module 190b can be configured to generate a depth image based on a structured light pattern. The CNN module 190b can be configured to perform various detection and / or recognition operations and / or perform 3D recognition operations.

[0123] CNN module 190b can be a dedicated hardware module configured to perform feature detection on video frames. Features detected by CNN module 190b can be used to compute descriptors. CNN module 190b can determine, in response to the descriptors, the probability that a pixel in a video frame belongs to a specific object and / or multiple objects. For example, using the descriptors, CNN module 190b can determine the probability that a pixel corresponds to a specific object (e.g., a person, furniture item, pet, vehicle, etc.) and / or characteristics of the object (e.g., the shape of the eyes, the distance between facial features, the hood of a vehicle, body parts, a vehicle's license plate, a person's face, clothing worn by a person, etc.). Implementing CNN module 190b as a dedicated hardware module of processor 102 allows device 100 to perform computer vision operations locally (e.g., on-chip) without relying on the processing power of remote devices (e.g., communicating data to a cloud computing service).

[0124] The computer vision operations performed by the CNN module 190b can be configured to perform feature detection on video frames to generate descriptors. The CNN module 190b can perform object detection to determine regions in the video frames with a high probability of matching a specific object. In one example, an open operand stack can be used to customize the type of the objects(s) to be matched (e.g., reference objects) (thus enabling the programmability of processor 102 to implement various artificial neural networks defined by directed acyclic graphs, each providing instructions for performing various types of object detection). The CNN module 190b can be configured to perform local masking on regions with a high probability of matching a specific object(s) to detect the object.

[0125] In some embodiments, the CNN module 190b can determine the location (e.g., 3D coordinates and / or positional coordinates) of various features (e.g., characteristics) of a detected object. In one example, 3D coordinates can be used to determine the position of a person's arms, legs, chest, and / or eyes. A positional coordinate for the vertical positioning of a body part in 3D space on a first axis and another coordinate for the horizontal positioning of a body part in 3D space on a second axis can be stored. In some embodiments, the distance from the lens 160 can represent a coordinate for the depth positioning of a body part in 3D space (e.g., a positional coordinate on a third axis). Using the positioning of the various body parts in 3D space, the processor 102 can determine the body position and / or the detected human body characteristics.

[0126] The CNN module 190b can be pre-trained (e.g., configured to perform computer vision to detect objects based on received training data). For example, the results of the training data (e.g., a machine learning model) can be pre-programmed and / or loaded into processor 102. The CNN module 190b can perform inference against the machine learning model (e.g., to perform object detection). Training may include determining weights for each layer of the neural network model. For example, weights can be determined for each layer used for feature extraction (e.g., convolutional layers) and / or for classification (e.g., fully connected layers). The weights learned by the CNN module 190b can vary depending on the design criteria of a particular implementation.

[0127] The CNN module 190b can perform feature extraction and / or object detection by performing convolution operations. These convolution operations can be hardware-accelerated for fast (e.g., real-time) computation that can be performed with low power consumption. In some embodiments, the convolution operations performed by the CNN module 190b can be used to perform computer vision operations. In some embodiments, the convolution operations performed by the CNN module 190b can be used for any function (e.g., 3D reconstruction) that may involve computing convolution operations and is performed by the processor 102.

[0128] Convolution operations can include sliding a feature detection window along a layer while performing computations (e.g., matrix operations). The feature detection window can apply filters to pixels and / or extract features associated with each layer. The feature detection window can be applied to a pixel and multiple surrounding pixels. In the example, a layer can be represented as a matrix representing the values ​​of features from one of the pixels and / or layers, and the filters applied by the feature detection window can be represented as matrices. Convolution operations can apply matrix multiplication between regions of the current layer covered by the feature detection window. Convolution operations can cause the feature detection window to slide along a region of the layer to generate a result representing each region. The size of the regions, the type of operation applied by the filters, and / or the number of layers can vary depending on the design criteria of a particular implementation.

[0129] Using convolutional operations, the CNN module 190b can compute multiple features for pixels of the input image at each extraction step. For example, each layer in a layer can receive input from a set of features located in a small neighborhood (e.g., a region) of a previous layer (e.g., a local receptive field). Convolutional operations can extract basic visual features (e.g., oriented edges, endpoints, corners, etc.), which are then combined by higher layers. Since the feature extraction window operates on pixels and nearby pixels (or subpixels), the result of the operation can have location invariance. Layers can include convolutional layers, pooling layers, non-linear layers, and / or fully connected layers. In the example, convolutional operations can learn to detect edges from raw pixels (e.g., the first layer), then use features from previous layers (e.g., detected edges) to detect shapes in the next layer, and then use the shapes to detect higher-level features in higher layers (e.g., facial features, pets, vehicles, vehicle components, furniture, etc.), and the last layer can be a classifier using the higher-level features.

[0130] The CNN module 190b can perform data streams directed to feature extraction and matching, including two-stage detection, warping operators, component operators manipulating component lists (e.g., components can be regions of vectors sharing common attributes and can be combined with bounding boxes), matrix inversion operators, dot product operators, convolution operators, conditional operators (e.g., multiplexing and demultiplexing), remapping operators, min-max reduction operators, pooling operators, non-minimum non-maximum suppression operators, scan-window-based non-maximum suppression operators, aggregation operators, scattering operators, statistical operators, classifier operators, integral image operators, comparison operators, indexing operators, pattern matching operators, feature extraction operators, feature detection operators, two-stage object detection operators, score generation operators, block reduction operators, and upsampling operators. The types of operations performed by the CNN module 190b to extract features from training data can vary depending on the design criteria of a particular implementation.

[0131] One or more of the hardware modules 190a-190n can be configured to implement other types of AI models. In one example, hardware modules 190a-190n can be configured to implement image-to-text AI models and / or video-to-text AI models. In another example, hardware modules 190a-190n can be configured to implement large language models (LLMs). Implementing AI models using hardware modules 190a-190n can provide AI acceleration, enabling the execution of complex AI tasks on edge devices such as edge devices 100a-100n.

[0132] One of the hardware modules 190a-190n can be configured to perform virtual aperture imaging. One of the hardware modules 190a-190n can be configured to perform transform operations (e.g., FFT, DCT, DFT, etc.). The number, type, and / or operations performed by the hardware modules 190a-190n can vary according to the design criteria of a particular implementation.

[0133] Each of the hardware modules 190a-190n can implement a processing resource (or hardware resource or hardware engine). Hardware engines 190a-190n can be operable to perform specific processing tasks. In some configurations, hardware engines 190a-190n can operate in parallel and independently of each other. In other configurations, hardware engines 190a-190n can operate collaboratively to perform assigned tasks. One or more of the hardware engines 190a-190n can be homogeneous processing resources (all circuits 190a-190n can have the same capabilities) or heterogeneous processing resources (two or more circuits 190a-190n can have different capabilities).

[0134] refer to Figure 5A block diagram illustrating an AI-adjusted region-of-interest (ROI) coding pipeline is shown. An AI-adjusted ROI coding pipeline 200 is illustrated. The AI-adjusted ROI coding pipeline 200 can be a representative example of an implementation on one of camera systems 100a-100n. The AI-adjusted ROI coding pipeline 200 may include a processor 102, a wireless communication device 156, input video frames 202a-202n, and / or output video frames 204a-204n. The AI-adjusted ROI coding pipeline 200 may include other components (not shown). The number, type, and / or arrangement of the components of the AI-adjusted ROI coding pipeline 200 can vary depending on the design criteria of a particular implementation.

[0135] Input video frames 202a-202n may include pixel data arranged as video frames. Input video frames 202a-202n may include raw (or uncompressed) video frames. For example, input video frames 202a-202n may be uncompressed video frames in YUV format. Uncompressed video frames 202a-202n may be generated by CMOS image sensor 180. Uncompressed video frames 202a-202n may be received by an interface of processor 102. For example, CMOS image sensor 180 may present a signal VIDEO including uncompressed video frames 202a-202n to the interface of processor 102. Each of the uncompressed video frames 202a-202n may include visual content, which may or may not include text.

[0136] Output video frames 204a-204n may include pixel data arranged as video frames. Output video frames 204a-204n may include encoded video frames. For example, output video frames 204a-204n may be generated in response to AI-adjusted ROI encoding performed by processor 102 on uncompressed video frames 202a-202n. For example, each of the encoded video frames 204a-204n may correspond to a corresponding one of the uncompressed video frames 202a-202n.

[0137] Encoded video frames 204a-204n may include fewer data bits compared to uncompressed video frames 202a-202n. For example, storing encoded video frames 204a-204n requires less storage capacity than storing uncompressed video frames 202a-202n, and / or transmitting encoded video frames 204a-204n requires less bandwidth than transmitting uncompressed video frames 202a-202n. Typically, the video quality of uncompressed video frames 202a-202n can be higher than that of encoded video frames 204a-204n. For example, AI-adjusted ROI encoding operations performed by processor 102 can preserve as much of the video quality of uncompressed video frames 202a-202n as possible while reducing the number of bits used to represent the same visual content. The compression used to generate encoded video frames 204a-204n can inherently reduce image quality compared to uncompressed video frames 202a-202n. Encoding uncompressed video frames 202a-202n can provide a trade-off between video quality and bitrate (e.g., file size). For example, a higher compression rate may result in lower video quality for encoded video frames 204a-204n. Compared to uncompressed video frames 202a-202n, AI-adjusted ROI encoding provided by processor 102 can generate encoded video frames 204a-204n with a reduced bitrate while maintaining text clarity in the video data.

[0138] The wireless communication module 156 can transmit encoded video frames 204a-204n. The wireless communication protocol and / or wireless communication channel available to the wireless communication module 156 may be bandwidth-limited. For example, transmitting uncompressed video frames 202a-202n via the wireless communication module 156 may be infeasible and / or may oversaturate the available bandwidth in the communication channel. The reduction in bit rate provided by the encoded video frames 204a-204n can enable the wireless communication module 156 to transmit the encoded video frames 204a-204n (e.g., within the bandwidth constraints of the communication channel).

[0139] In some embodiments, wireless transmission of the encoded video frames 204a-204n may be presented to a cloud storage service and / or a remote computing device. In one example, the remote computing device may have limited storage capacity. In another example, the cloud storage service may provide charged, high-capacity data storage, where the higher the amount of data stored in the cloud storage service, the higher the fee. In some embodiments, the encoded video frames 204a-204n may be stored locally on the respective camera systems 100a-100n. For example, the camera systems 100a-100n may implement a local storage device (e.g., a microSD card) with limited storage capacity (e.g., providing loop recording, where the latest data can overwrite the oldest data when the storage device is full). Storing the encoded video frames 204a-204n at a specific average bit rate (e.g., a lower bit rate compared to the uncompressed video frames 202a-202n) allows more data to be stored on a particular storage medium. The amount of available storage capacity and / or the cost associated with storing the encoded video frames 204a-204n may vary depending on the design criteria of the particular implementation.

[0140] Processor 102 may include blocks (or circuits) 210, 212, 214, and / or 216. Circuit 210 may implement a video preprocessing pipeline. Circuit 212 may implement an object (or vehicle) detection CNN. Circuit 214 may implement a text location detection CNN. Circuit 216 may implement a video encoding module. Each of circuits 210-216 may be implemented as a video encoding module. Figure 4 A combination of one or more of the hardware modules 190a-190n shown in association. Processor 102 may include other components (not shown). The number, type, and / or arrangement of the components of processor 102 used to implement AI-adjusted ROI encoding may vary according to the design criteria of a particular implementation.

[0141] Video preprocessing pipeline 210 can be configured to receive a signal VIDEO. Video preprocessing pipeline 210 can be configured to generate a signal (e.g., PVID) and / or a signal (e.g., DVID) in response to the signal VIDEO. The signal PVID may include preprocessed video data. Video preprocessing pipeline 210 can be configured to perform various preprocessing operations on uncompressed video frames 202a-202n. Video preprocessing pipeline 210 can be configured to present the signal PVID to video encoding module 216. In some embodiments, video preprocessing pipeline 210 can be configured to present the signal PVID to memory 150 (e.g., for storage). In some embodiments, video preprocessing pipeline 210 can be configured to pass the signal PVID to other components of processor 102 for other types of video processing.

[0142] Video preprocessing pipeline 210 can be configured to receive pixel data from the signal VIDEO. Video preprocessing pipeline 210 can be configured to process pixel data arranged into video frames. The preprocessing performed by preprocessing pipeline 210 can prepare the video data using various preprocessing operations (e.g., motion detection, cropping, auto-balancing, trimming, stabilization, scaling up, scaling down, dedistorting, formatting for output devices, color space conversion, noise reduction, etc.) that can be used for various types of analysis (e.g., object detection, action detection, depth analysis, object tracking, etc.). Video preprocessing pipeline 210 can be configured to prepare raw pixel data in uncompressed video frames 202a-202n for further analysis by a neural network implemented by AI-adjusted region-of-interest coding pipeline 200. The preprocessed video frames in the signal PVID may include full-size versions of the input video frames 202a-202n.

[0143] The video preprocessing pipeline 210 may include a block (or circuit) 220. Circuit 220 may implement a downscaling module. Downscaling module 220 may be configured to generate a signal DVID in response to a signal VIDEO. In some embodiments, signal DVID may include downscaled versions of input video frames 202a-202n. In some embodiments, signal DVID may include cropped versions of input video frames 202a-202n. Downscaling module 220 may be configured to generate video frames that may be smaller versions of the input video frames 202a-202n. For example, the video data presented in signal DVID may include a lower resolution compared to the resolution of the input video frames 202a-202n and / or the preprocessed video frames in signal DVID. In one example, downscaling module 220 may be configured to perform a downscaling operation (e.g., reducing the resolution of input video frames 202a-202n by scaling the video data to a proportionally smaller size). In another example, the reduction module 220 can be configured to perform a cropping operation (e.g., reducing the resolution of input video frames 202a-202n by removing a portion of the input video frames 202a-202n). For example, video data near the top of the input video frames 202a-202n can be cropped because vehicles are unlikely to appear near the top of the video frames (e.g., in the sky). The type of operation performed to reduce the resolution of video data in the signal DVID can vary depending on the design criteria of the specific implementation.

[0144] The signal DVID can be presented to the object detection CNN 212. For example, performing video detection operations (e.g., vehicle detection, sign detection, license plate detection, text detection) on video data at a lower resolution than the input video frames 202a-202n allows specific objects to still be detected while reducing the amount of computation performed. For example, performing object detection on video data with a smaller size than the full-size input video frames 202a-202n can consume fewer resources, reduce power consumption, and / or reduce computation time. The data coordinates determined in response to object detection can be mapped to coordinates in the full-size video frames (e.g., the input video frames 202a-202n and / or the preprocessed video frames in the signal PVID).

[0145] The object detection CNN 212 can implement a lightweight neural network. The object detection CNN 212 can be configured to receive a signal DVID. In some embodiments, the object detection CNN 212 can receive a signal PVID (e.g., object detection can be performed on full-size preprocessed video frames). The object detection CNN 212 can include a block (or circuit) 222. Circuit 222 can implement a neural network model. The object detection CNN 212 can be configured to implement a neural network model 222 trained to recognize the position and / or size of a vehicle in uncompressed, preprocessed, and / or downsized video frames. For example, the neural network model 222 implemented by the object detection CNN 212 can be trained on labeled training data to provide an indication of vehicle position in response to video data. In some embodiments, the neural network model 222 can be trained to detect the position of road signs. The object detection CNN 212 can be configured to perform a full-frame search for vehicles. In some embodiments, the object detection CNN 212 can be configured to detect vehicles in specific regions of video data (e.g., the vehicle search can be limited to road regions in the video frames). Object detection CNN 212 can generate bounding box locations of vehicles detected in uncompressed video frames 202a-202n. Object detection CNN 212 can be configured to generate a signal (e.g., VLOC) in response to a signal DVID. The signal VLOC can be presented to text location detection CNN 214. The signal DVID can be passed to text location detection CNN 214.

[0146] The signal VLOC can include vehicle position data. In one example, the vehicle position data can include one or more coordinates of a video frame, as well as height and width data for each detected vehicle. In another example, the vehicle position data can include the four corners of each bounding box. In yet another example, the vehicle position data can include a list of coded blocks of regions corresponding to each detected vehicle. The vehicle position data can provide bounding box information for each vehicle detected in one or more video frames provided by the signal DVID. For example, after a vehicle position (or sign position) is detected, the bounding box parameters (e.g., rendered in the signal VLOC) and the scaled-down full video frame (e.g., rendered in the signal DVID) can be sent to the text position detection CNN 214. The format of the vehicle position data can vary depending on the design criteria of a particular implementation.

[0147] The text location detection CNN 214 can implement a lightweight neural network. The text location detection CNN 214 can be configured to receive signals VLOC and / or DVID. The text location detection CNN 214 can include a block (or circuit) 224. Circuit 224 can implement a neural network model. The text location detection CNN 214 can be configured to implement a neural network model 224 trained to recognize the location of a license plate within a vehicle bounding box in an uncompressed, preprocessed, and / or downsized video frame. For example, the neural network model 224 implemented by the text location detection CNN 214 can be trained on labeled training data to provide indications of license plate location and / or license plate size in response to video data and / or bounding box information for the vehicle. In some embodiments, the neural network model 224 can be trained to detect text on road signs.

[0148] The text location detection CNN 214 can be configured to limit the search region to the bounding box locations of vehicles and / or signs in a video frame. For example, the text location detection CNN 214 can be configured to detect license plates in less than the entire video frame. Limiting the license plate detection search to the location of the vehicle bounding box can limit the amount of computational resources used (e.g., less data to analyze compared to the entire video frame) and / or prevent false positives (e.g., avoiding the detection of objects that look like license plates and / or decorative license plates that may be mounted on a wall). The text location detection CNN 214 can generate regions of interest corresponding to the locations of the license plates of vehicles detected in uncompressed video frames 202a-202n. The text location detection CNN 214 can be configured to generate a signal (e.g., TLOC) in response to a signal VLOC. The signal TLOC can be presented to the video encoding module 216.

[0149] The signal TLOC can include region of interest (ROI) data. In one example, the ROI data may include one or more coordinates of a video frame, as well as height and width data for each detected license plate. In another example, the ROI data may include the four corners of each license plate bounding box. In yet another example, the ROI data may include a list of coded blocks corresponding to the region of each detected license plate. The ROI data can provide bounding box information for license plates detected within one or more vehicle bounding boxes provided by the signal VLOC. The format of the vehicle position data may vary depending on the design standards of the specific implementation.

[0150] In the illustrated example of the AI-adjusted region of interest (ROI) encoding pipeline 200, two lightweight CNNs can be implemented for license plate detection. In some embodiments, a single CNN can be implemented to perform both vehicle detection and license plate detection. In some embodiments, other types of vehicle and / or license plate detection (e.g., feature extraction and / or object detection) can be performed without implementing a CNN. The type of video analysis performed by processor 102 to detect vehicle bounding boxes and / or license plate ROIs can vary depending on the design criteria of the particular implementation.

[0151] Video encoding module 216 can be configured to perform video compression and / or encoding operations. Video encoding module 216 can be configured to receive a signal TLOC and / or a signal PVID. Video encoding module 216 can be configured to receive pixel data arranged as video frames during real-time generation of video data. The signal TLOC may include coordinates of text positions (e.g., license plate coordinates). Video encoding module 216 can be configured to map the coordinates of the license plate and / or text positions to full-size video data in the signal PVID. The pre-processed full-size video data of the video frames in the signal PVID can be used for video encoding. Video encoding module 216 can be configured to generate a signal (e.g., TEVID) in response to the signal TLOC and signal PVID. The signal TEVID may include encoded video frames 204a-204n. The signal TEVID may be presented to wireless communication module 156. In some embodiments, the signal TEVID may be stored locally (e.g., via memory 150).

[0152] Video encoding module 216 can be configured to compress video data. In one example, the compression performed by video encoding module 216 may be H.264 encoding. In another example, the compression performed by video encoding module 216 may be H.265 encoding. In yet another example, the compression performed by video encoding module 216 may be AV1 encoding. For example, video encoding module 216 is capable of generating 4K ultra-high resolution using H.264 encoding at twice the real-time speed (e.g., 60fps). In another example, video encoding module 216 is capable of generating 4K ultra-high resolution (e.g., 4KP30 AVC and HEVC encoding with multi-stream support) using H.265 / HEVC and / or 4K AVC encoding at 60fps. Video encoding module 216 can be configured to convert raw, uncompressed video streams into specific digital formats suitable for storage, transmission, and / or playback. Video encoding module 216 can be configured to apply video codecs (e.g., H.264, H.265, VP9, ​​AV1, etc.) to compress video data. The video encoding module 216 can be configured to multiplex compressed video and compressed audio into a container format (e.g., MP4, MKV). The video encoding module 216 can be configured to add metadata to the video data (e.g., camera ID, camera brand / model, GPS data, resolution, bitrate, frame rate, etc.). The encoding operations performed by the video encoding module 216 can vary depending on the design standards of a particular implementation.

[0153] For bandwidth-constrained and / or resource-constrained operation of the AI-adjusted Region of Interest (ROI) coding pipeline 200, the video coding module 216 can be configured to implement multiple video coding parameters. In one example, the video coding module 216 can be configured to implement at least two sets of video coding parameters 226-228. A set of video coding parameters 226 (e.g., general coding parameters) can be selected for locations in uncompressed video frames 202a-202n that do not contain ROIs (e.g., license plate text and / or other types of text, e.g., road sign text). For example, in uncompressed video frames 202a-202n where no vehicles and / or license plates are detected, the same set of general video coding parameters 226 can be used to encode the entire uncompressed video frame. The general coding parameters 226 can be selected to achieve a target average bit rate for the encoded video frames 204a-204n. For example, the target average bit rate can be approximated by the available bandwidth of the communication channel used by the wireless communication device 156. A second set of encoding parameters 228 (e.g., text clarity parameters) can be selected for the locations including Regions of Interest (ROIs) in uncompressed video frames 202a-202n. For example, in uncompressed video frames 202a-202n where vehicles and / or license plates are detected, the ROIs can be encoded using text clarity parameters 228, and regions outside the ROIs in uncompressed video frames 202a-202n can be encoded using general encoding parameters 226 (or other lower-quality video parameters).

[0154] Video encoding module 216 can be configured to determine an offset value to be applied to general encoding parameter 226. The offset value can be a negative offset value. A negative offset value can be used to select text clarity parameter 228. For example, a lower QP can result in higher quality video and / or text clarity. In some embodiments, video encoding module 216 can be configured to add a positive offset value to general encoding parameter 226. Because a negative offset value for text clarity parameter 228 can increase the bitrate of encoded video frames 204a-204n, a positive offset applied to general encoding parameter 226 can be compensated for by reducing the bitrate to achieve the original target bitrate (e.g., the bitrate when no text is detected). Typically, since the ROI for the license plate location may comprise a relatively small portion of the video frame, the amount of positive offset for general encoding parameter 226 may be less than the negative offset applied to select text clarity parameter 228.

[0155] The signal TLOC may include data for the ROI. The ROI may be used to determine encoding parameters (e.g., QPs in macroblocks for general encoding parameters 226 and / or text clarity parameters 228). For example, a QP may be a type of weight. A QP may be an encoding tool that can influence how the video encoding module 216 allocates H.264 / H.265 encoded bits to encoded video frames in the signal TEVID. In some embodiments, the encoding parameters may be determined in response to a combination of the overall size of the ROI, the distance of the ROI from the image sensor 180, the relative velocity of the detected ROI with respect to the image sensor 180, etc.

[0156] In some embodiments, the object detection CNN 212 can determine the distance of a detected vehicle from the image sensor 180. The distance can be determined based on the relative size of the detected vehicle. The text location detection CNN 214 can be configured to apply filters to the license plate detection location. The filters can remove license plates determined to be too far to provide clear text. For example, license plates that may be considered too far away may already have unreadable text in the source uncompressed video frames 202a-202n. In the remaining license plate ROIs (e.g., license plates within a predetermined distance for text clarity), the video encoding module 216 can independently select a QP for each license plate. For example, the text clarity encoding parameter 228 can be adaptively selected for each license plate based on distance. Generally, text located further away may be more difficult to read and / or may be more negatively affected by encoding. For example, a larger negative offset value can be selected for the text clarity encoding parameter 228 for license plates that are further away, and a smaller negative offset value can be selected for the text clarity encoding parameter 228 for license plates that are closer to the image sensor 180. Multiple sets of text clarity encoding parameters 228 can be determined to provide text clarity for multiple license plates detected in the same video frame. The negative offset applied for each distance can vary according to the design criteria of a particular implementation.

[0157] refer to Figure 6The diagram illustrates the performance of computer vision operations on an example video frame to detect the location of a vehicle bounding box. An example video frame 250 is shown. Example video frame 250 may include pixel data captured by capture device 104. In one example, video frame 250 may be provided to processor 102 as a signal VIDEO. In another example, video frame 250 may be generated by processor 102 in response to pixel data provided in the signal VIDEO. The pixel data of video frame 250 received by processor 102 may correspond to one of uncompressed video frames 202a-202n. In some embodiments, video frame 250 may be a preprocessed video frame provided by signal PVID. In some embodiments, video frame 240 may be a downsized video frame provided by signal DVID. In some embodiments, example video frame 250 may be presented as human-visual video output to one or more video displays. In some embodiments, example video frame 250 may be used internally by processor 102 to perform computer vision operations. For example, video frame 250 may be analyzed by object detection CNN 212.

[0158] Example video frame 250 may include a view of road 252. In the example, example video frame 250 may be captured by one of a camera system 100a-100n mounted to vehicle 80 (e.g., a view provided by a full-around view 92a-92d). A portion of vehicle 80 traveling on road 252 is shown in video frame 250. Road 252 may include lanes 254a-254d. An elevated road 256 is shown above road 252. For example, the external environment 70 shown in video frame 250 may include a highway system. Vehicles 258a-258c are shown. Vehicles 258a-258c may be on road 252 in front of vehicle 80. Vehicle 258a may be in lane 254d, and vehicle 258c may be in lane 254c. Each of vehicles 258a-258c may have a corresponding license plate 260a-260c. In the example shown, license plate 260a may include the characters "ABC123". Signs 262-268 are shown. Sign 262 may be an overhead directional road sign, sign 264 may be a distant sign, sign 266 may be an advertisement, and sign 268 may be a speed limit road sign.

[0159] Dashed shapes 270a-270c are shown in video frame 250. Dashed shapes 270a-270c can represent the detection of objects / subjects by computer vision operations performed by processor 102. Dashed shapes 270a-270c can include pixel data corresponding to objects detected by AI-adjusted region of interest coding pipeline 200, neural network model 190b, and / or video-to-text AI model. For AI-adjusted region of interest coding pipeline 200, dashed shapes 270a-270c can correspond to objects detected by object detection CNN 212. In the example shown, dashed shape 270a can correspond to vehicle 258a, dashed shape 270b can correspond to vehicle 258b, and dashed shape 270c can correspond to vehicle 258c. For illustrative purposes, only dashed shapes 270a-270c are shown. However, other types of objects (e.g., signs 262-268, pedestrians, bicycles, lane dividers, etc.) can be detected as objects. In some embodiments, various other types of objects can be detected in response to animal detection, household object detection, interior object detection, person detection, vehicle detection, road detection, sky region detection, obstacle detection, and / or exterior object detection (e.g., one or more of neural network 190b, video-to-text AI models, object detection CNN 212, and / or text location detection CNN 214 may include libraries configured to detect people, vehicles, objects, animals, etc.). In the illustrated example, the implemented library and / or training data used to train the AI ​​model (e.g., neural network model 222 and / or neural network model 224) can be configured to perform detection and / or description of objects (e.g., vehicles) that may include text. For example, the implemented library may be configured to detect cars, vans, trucks, SUVs, motorcycles, delivery vans, transport trucks, long-haul vehicles, construction vehicles, etc. Dashed shapes 270a-270c are shown for illustrative purposes. In the example, dashed shapes 270a-270c can be a visual representation of object detection (e.g., dashed shapes 270a-270c may not appear on the output video frame in the signal VIDOUT and / or video TEVID). In another example, dashed shapes 270a-270c can be bounding boxes generated by processor 102 and displayed on the output video frame to indicate that an object has been detected (e.g., dashed shapes 270a-270c may be displayed in debug operation mode).

[0160] Computer vision operations, vehicle detection and analysis, license plate detection, and / or video-to-text (or sensor fusion to text) operations can be configured to detect characteristics of detected objects, behavior of detected objects, direction of movement of detected objects, context of detected objects, and / or presence of detected objects. Object characteristics may include height, length, width, slope, arc length, color, color temperature, emitted light intensity, detected text on the object, movement path, movement speed, direction of movement, proximity to other objects, etc. Detected object characteristics may include the object's state (e.g., open, closed, on, off, etc.). Detected object characteristics may include distance measurements from lens 160 to the detected object. Behavior and / or presence can be determined in response to the object's type and / or the characteristics of the detected object. While an example video frame 250 is shown, object behavior, direction of movement (e.g., trajectory), and / or presence can be determined by analyzing a sequence of video frames captured over time. For example, movement path and / or speed characteristics can be used to determine whether an object classified as a person is walking or running. The speed and / or direction of movement can be used to track the position of an object across multiple video frames and / or estimate its position between video frames and / or between multiple video frame intervals. The type of detected characteristics and / or behaviors can vary depending on the design criteria of a particular implementation.

[0161] In the example shown, bounding boxes 270a-270c can be regions of interest for a subset of objects in video frame 250. Bounding boxes 270a-270c are shown as representative examples of various objects, but typically many more objects (e.g., dents, scratches, animals, other people, etc.) can be detected. In the example, the settings (e.g., feature sets) for processor 102 (e.g., a computer vision AI neural network model implemented by neural network 190b, a video-to-text AI model, an object detection CNN 212, and / or a text location detection CNN 214) can define objects of interest as vehicles, pets, people, storage objects, sports equipment, tools, supplies, camera obstructions, etc. For example, for a feature set defined as detecting objects inside or near a vehicle, porches, ceilings, and / or stairs might not be objects of interest. In the example shown, bounding boxes 270a-270c are shown as having a cube (or rectangle) shape. In some embodiments, the shape of the bounding box 270a-270c corresponding to the detected object of interest can be formed to follow the shape of the body of the detected vehicle and / or the shape of various detected objects (e.g., following the curves and / or irregular shapes of the body of the detected objects).

[0162] Processor 102, CNN module 190b, and / or video-to-text AI model can be configured to implement region, vehicle, road sign, animal, lens obstacle, object, and / or face detection techniques. In some embodiments, other types of subjects (e.g., passengers, pedestrians, street signs, etc.) can be detected as objects of interest. Computer vision techniques and / or video-to-text techniques can be configured to detect regions of interest (ROIs) of the detected objects 270a-270c and / or generally generate contextual information about the detected objects 270a-270c and / or the scene. For example, bounding boxes 270a-270c can be visual representations of the detected ROIs. Computer vision techniques can be looped (e.g., to perform object / subject detection iteratively throughout example video frame 250) to determine whether any object of interest (e.g., as defined by a feature set) is within the field of view of lens 160 and / or image sensor 180.

[0163] Although only objects 270a-270c are shown as objects of interest (e.g., vehicles 258a-258c), computer vision operations and / or video-to-text operations performed by processor 102, neural network 190b, video-to-text AI model, object detection CNN 212, and / or text location detection CNN 214 can be configured to detect background objects and / or other types of objects. Background objects can be detected for other computer vision purposes (e.g., training data, labeling, depth detection, etc.). The type of subject identified as objects of interest 270a-270c can vary depending on the design criteria of the specific implementation. Details of computer vision, video-to-text manipulation, and / or sensor fusion to text manipulation can be described in connection with the following patent applications: U.S. Patent Application No. 18 / 583,298, filed February 11, 2024; U.S. Patent Application No. 18 / 621,504, filed March 29, 2024; U.S. Patent Application No. 18 / 657,588, filed May 7, 2024; and / or U.S. Patent Application No. 18 / 657,492, filed May 7, 2024, the appropriate portions of which are incorporated herein by reference.

[0164] Bounding boxes 270a-270c can represent the positions of vehicles 258a-258c in video frame 250. For example, bounding boxes 270a-270c can be determined by object detection CNN 212. Bounding boxes 270a-270c can be provided to text location detection CNN 214 in the signal VLOC. Text location detection CNN 214 can use the vehicle positions of bounding boxes 270a-270c to detect the positions of license plates 260a-260c (which will be compared with...). Figure 7(Described in association). In the example shown, vehicle 258a could be a sedan, vehicle 258b could be a minivan, and vehicle 258c could be a sedan. The types of vehicles 258a-258c detected for bounding boxes 270a-270c are representative examples. Other types of vehicles can be detected (e.g., trucks, SUVs, hatchbacks, crossovers, coupes, sports cars, station wagons, convertibles, dump trucks, transport trucks, concrete mixers, garbage trucks, ambulances, fire trucks, flatbed trucks, agricultural vehicles, etc.). The types of vehicles detected by the object detection CNN 212 can vary depending on the design criteria of a particular implementation.

[0165] A dashed arrow (e.g., DA-DC) is shown. The dashed arrow DA-DC may correspond to one of the bounding boxes 270a-270c. The dashed arrow DA-DC may represent the distance of bounding boxes 270a-270c from image sensor 180. The object detection CNN 212 can be configured to determine the distance from each detected object to image sensor 180. In the example shown, distance DA may be the calculated distance from image sensor 180 to bounding box 270a (e.g., vehicle position for vehicle 258a), distance DB may be the calculated distance from image sensor 180 to bounding box 270b (e.g., vehicle position for vehicle 258b), and distance DC may be the calculated distance from image sensor 180 to bounding box 270c (e.g., vehicle position for vehicle 258c).

[0166] Encoding module 216 can be configured to select different QP settings for each license plate and / or for each of the texts of detected objects. In one example, the QP settings can be selected based on the size of the license plate and / or the size of the text. Typically, the sizes of vehicles 258a-258c can be similar. For example, most vehicles, except motorcycles, can have license plates of the same size. Since the vehicle sizes can be similar, the size of each license plate can typically be proportional to the size of the bounding boxes 270a-270c. The size of the bounding boxes 270a-270c can typically be proportional to the distance DA-DC. For example, object detection CNN 212, text location detection CNN 214, and / or video encoding module 216 can use the distance DA-DC to determine the QP settings and / or which objects to skip for text augmentation.

[0167] refer to Figure 7 The diagram illustrates vehicle license plate detection at the block level of a video frame. An example license plate detection 300 is shown. License plate detection 300 may include, for example, […]. Figure 6An illustrative example of video frame 250 described in connection with this description is shown. For example, ego vehicle 80, road 252, and vehicles 258a-258c are shown. License plate detection 300 can represent the determination of the ROI performed by text location detection CNN 214 in response to the signal VLOC and uncompressed video frames 202a-202n.

[0168] The license plate detection 300 may include multiple vertical lines 302a-302m and multiple horizontal lines 304a-304l. The vertical lines 302a-302m and the horizontal lines 304a-304l may form a grid pattern. The grid pattern may include multiple blocks 306aa-306mn. The grid pattern 306aa-306mn may represent the location of encoded blocks for each of the uncompressed video frames 202a-202n. The number of encoded blocks 306aa-306mn for each of the uncompressed video frames 202a-202n may depend on the size (e.g., resolution) of the uncompressed video frames (or the downsized video frames in the DVID signal) and / or the size of the encoded blocks 306aa-306mn. In some embodiments, the size of the encoded blocks 306aa-306mn may be a variable value ranging from 4x4 pixels to 64x64 pixels. In one example, coding blocks 306aa-306mn can each be a 16x16 pixel CTU (Coding Tree Unit) for the H.265 coding standard. Typically, coding blocks 306aa-306mn can be rectangular in shape, with their pixel width / height being a power of 2. For example, due to the rectangular shape of coding blocks 306aa-306mn, processor 102 can be configured to map (e.g., license plate parameters) any bounding box to the rectangle of coding blocks 306aa-306mn in H.264 or H.265 (e.g., by rounding up). The size of each of the coding blocks 306aa-306mn can vary depending on the design criteria of a particular implementation.

[0169] Vehicle positions 310a-310b are shown. Vehicle positions 310a-310b can correspond to... Figure 6 The bounding boxes 270a-270b are shown in association. For example, vehicle position 310a may correspond to vehicle 258a, and vehicle position 310b may correspond to vehicle 258b. Vehicle 258c (using with...) Figure 6 The bounding box 270c shown in association (detected) may not have a corresponding vehicle location in license plate detection 300. For example, processor 102, object detection CNN 212, and / or text location detection CNN 214 may have already filtered out vehicles 258c based on distance DC. For example, license plate 260c may have been determined to be too far away to achieve readable text (e.g., the text may already be unreadable in the source uncompressed video frame, and adjusting the encoding parameters may not provide any benefit).

[0170] Shaded areas 312a-312b are shown. Shaded areas 312a-312b may correspond to the positions of license plates 260a-260b within the corresponding vehicle positions 310a-310b. Shaded areas 312a-312b may represent detected license plate ROIs. License plate ROIs 312a-312b can be detected by a text location detection CNN 214 in response to a signal VLOC. The signal TLOC may include license plate ROIs 312a-312b. Although two of the license plate ROIs 312a-312b are shown in the example license plate detection 300, the number of detected license plate ROIs may vary based on the number of vehicles detected in uncompressed video frames 202a-202n and / or the distance to the vehicles.

[0171] License plate ROIs 312a-312b may include a total coded block area (e.g., a macroblock / CTB area). License plate ROIs 312a-312b may include one or more coded blocks 306aa-306mn. For example, license plate ROIs 312a-312b may correspond to complete coded blocks 306aa-306mn. Complete coded blocks may be selected for license plate ROIs 312a-312b even if the detected corresponding license plate text is only a portion of one or more coded blocks 306aa-306mn. The number of coded blocks 306aa-306mn within each of license plate ROIs 312a-312b may depend on the size of the detected license plate. In the example shown, license plate ROI 312a may include six coded blocks from coded blocks 306aa-306mn, and license plate ROI 312b may include two coded blocks from coded blocks 306aa-306mn. The license plate ROIs 312a-312b may include a number of squares in the coded blocks 306aa-306mn that are not smaller than the bounding boxes detected for the license plates 260a-260b. The bounding box for the license plate ROIs 312a-312b may be a rectangle for the vehicle license plate, having the size of the pixel height and width of the license plate in the uncompressed video frames 202a-202n.

[0172] Dashed circles 320a-320d are shown at the corners of license plate ROI 312a. These dashed circles 320a-320d can represent the corner pixels of license plate ROI 312a. For example, the bounding box for license plate 260a detected by the text location detection CNN 214 can be defined by corner pixels 320a-320d. For example, the number of pixels from corner pixel 320a to corner pixel 320d can be the width of the bounding box for license plate ROI 312a, and the number of pixels from corner pixel 320a to corner pixel 320b can be the height of the bounding box for license plate ROI 312a. For example, if each of the coded blocks 306aa-306mn is a 16x16 pixel block, then license plate ROI 312a could be 48 pixels wide and 32 pixels high. Although corner pixels 320a-320d are shown as an illustrative example for license plate ROI 312a only, corner pixels can similarly represent the size of the bounding box for license plate ROI 312b and / or other detected license plates.

[0173] The text location detection CNN 214 can be configured to perform license plate detection within vehicle locations 310a-310b. For example, license plate detection can be limited to vehicle locations 310a-310b and may not be performed outside of them. Limiting license plate detection to vehicle locations 310a-310b provides efficient use of computational resources (e.g., power consumption, processing cycles, etc.). For example, computational resources may not be wasted trying to detect a license plate where it shouldn't be. Limiting license plate detection to vehicle locations 310a-310b further prevents false positives. For example, a license plate not within a vehicle location could be a false positive (e.g., a decorative license plate that may be hung on a wall or side of a building, a discarded license plate, etc.). Limiting the operation performed by the text location detection CNN 214 to vehicle locations 310a-310b provides access to a smaller portion of the video frame compared to performing the operation on the entire video frame.

[0174] In some embodiments, the object detection CNN 212 and / or the text location detection CNN 214 can be configured to track the vehicle location (e.g., bounding boxes 270a-270c and / or vehicle locations 310a-310b) of detected objects over time. In some embodiments, the object detection CNN 212 and / or the text location detection CNN 214 can be configured to determine the location of bounding boxes 270a-270c and / or license plate ROIs 312a-312b in each of uncompressed video frames 202a-202n (or downsized video frames in the signal DVID). In some embodiments, the object detection CNN 212 and / or the text location detection CNN 214 can be configured to perform vehicle detection at predetermined intervals. In the example, the processor 102, the object detection CNN 212, and / or the text location detection CNN 214 can implement tracking to predict the location of vehicles 258a-258c and / or license plates 260a-260c between detection intervals. For example, tracking can be performed based on the distance, direction of travel, and / or relative speed of vehicles 258a-258c determined at each detection interval, until the next detection interval. Predictive tracking of the location of license plate ROIs 312a-312b between regular detection intervals can save AI computational resources. In one example, the predefined detection interval could be once per specific amount of time (e.g., once every 10 seconds, once every 20 seconds, once every 30 seconds, etc.). In another example, the predefined detection interval could be once per specific number of video frames (e.g., once every two frames, once every three frames, once every five frames, etc.). In yet another example, the detection interval can be adapted based on the amount of movement of the ego vehicle 80 and / or traffic speed. Typically, tracking can be performed as part of vehicle detection performed by an object detection CNN 212 and / or license plate detection performed by a text location detection CNN 214. For example, object tracking can be optimized for frames per second and bounding box accuracy using temporal information. The amount of time between detection intervals and / or the method for selecting detection intervals can vary depending on the design criteria of a particular implementation.

[0175] For each video frame (or detection interval), the text location detection CNN 214 can apply filtering to each of the license plate bounding boxes and / or the bounding boxes of the detected vehicles 270a-270c. The filtering can be configured to remove (or ignore) license plates that may be too far from the vehicle 80 (or image sensor 180) to provide legible text. For example, a distant license plate may include a license plate that might be unreadable even in uncompressed video frames 202a-202n. Encoding a license plate that initially does not have legible text may not provide better clarity in the encoded video frames 204a-204n. In the example shown, the distance DC to vehicle 258c may be too far to provide legible text, and the text location detection CNN 214 can filter out license plate 260c to generate a license plate ROI. For each of the remaining license plate ROIs (e.g., license plate ROIs 312a-312b in the example shown), the processor 102 can determine an adaptive QP offset value. For example, over time, if vehicle 258c moves closer to image sensor 180, a license plate ROI can be determined for license plate 260c. Similarly, over time, if vehicle 258a and / or vehicle 258b moves further away from image sensor 180, license plate ROIs 312a-312b can be filtered out. In one example, the distance can be a predetermined distance value. For example, the predetermined distance value can be determined in response to engineering experience and / or limitations of image sensor 180. In some embodiments, the distance threshold can be reduced based on weather conditions (e.g., foggy weather may have reduced visibility). The specific distance used to filter out license plates can depend on the resolution of image sensor 180 and / or can vary according to the design criteria of a particular implementation.

[0176] In some embodiments, the tracking performed by the text location detection CNN 214 can be configured to detect the relative speeds of vehicles 258a-258c. The text location detection CNN 214 can be configured to determine various parameters of the image sensor 180 (e.g., sensor gain level, exposure length, lens distortion, etc.). The combination of relative speeds and / or parameters of the image sensor 180 can be used to determine the noise level in the video frames and / or the motion blur (e.g., distortion level) of the vehicles in the uncompressed video frames 202a-202n. The noise level and / or motion blur can be used by the video encoding module 216 to determine an adaptive QP offset for each of the license plate ROIs 312a-312b.

[0177] The QP setting for license plate ROIs 312a-312b can be adaptively selected. In one example, the QP setting can be selected based on the distance DA-DC (e.g., license plate size). In another example, the QP setting can be selected based on the relative velocity, motion blur, and / or noise (e.g., distortion level) determined for license plate ROIs 312a-312b. In yet another example, the QP setting can be determined based on a combination of distance and / or relative velocity. The QP setting can be adaptively selected to provide legible text (if possible) for each of license plates 260a-260c. Different QP settings can be applied to each of license plate ROIs 312a-312b. An adaptive QP offset can be selected for each license plate ROI to provide a balance between text sharpness and overall video bitrate. For example, background detail in encoded video frames 204a-204n can be sacrificed (e.g., a higher QP setting) to save bits for sharper vehicle license plate text (e.g., a lower QP setting).

[0178] refer to Figure 8 The diagram illustrates an example encoding parameter offset to be applied to the region of interest on a license plate. An example adaptive encoding parameter offset 350 is shown. The example adaptive encoding parameter offset 350 may include an encoding region 352. In the example shown, encoding region 352 may correspond to... Figure 7 The license plate ROI 312a is shown in association. The video encoding module 216 can be configured to apply an adaptive encoding parameter offset of 350 to the uncompressed video frames 202a-202n for the license plate ROIs 312a-312b to provide enhanced text clarity in the encoded video frames 204a-204n.

[0179] The encoding region 352 may include multiple offset parameters 354aa-354bc. The offset parameters 354aa-354bc can correspond to... Figure 7A subset of coded blocks 306aa-306mn is shown in association. This subset of coded blocks 306aa-306mn may correspond to a coded block within the bounding box of one of the license plates for license plate ROIs 312a-312b. In the example shown, offset parameters 354aa-354bc may correspond to coded blocks 306aa-306mn of license plate ROI 312a. For example, license plate ROI 312a may include six coded blocks from coded blocks 306aa-306mn, which may have six offset parameters from the corresponding offset parameters 354aa-354bc. Different license plate ROIs may include different numbers and / or positions of coded blocks 306aa-306mn in uncompressed video frames 202a-202n, resulting in a corresponding number of offset parameters. The number of adjusted offset parameters for each detected license plate may vary depending on the design criteria of the specific implementation.

[0180] Encoding parameters can be selected by the video encoding module 216. These encoding parameters can be of one type used to generate encoded video frames 204a-204n. The encoding parameters adjusted by the video encoding module 216 can be adjusted in real-time (e.g., frame-by-frame). In one example, if the selected encoding protocol is H.264, the encoding parameters can be applied to macroblocks. In another example, if the selected encoding protocol is H.265, the encoding parameters can be applied to CTBs. Encoding parameters can be selected for encoding blocks 306aa-306mn of uncompressed video frames 202a-202n. Various encoding parameters that the video encoding module 216 can apply to encoding blocks 306aa-306mn can exist. In one example, the encoding parameters adjusted for license plate ROIs 312a-312b can be QP. For example, video encoding module 216 can select offset parameters 354aa-354bc to adjust QP frame-by-frame for encoding region 352 (e.g., providing an offset from general encoding parameter 226) to apply text clarity encoding parameter 228.

[0181] Video encoding module 216 can be configured to generate encoded video frames 204a-204n at a predefined average bitrate. For example, video encoding module 216 can be configured to apply general encoding parameters 226 to uncompressed video frames 202a-202n to generate a target average bitrate for the encoded video frames 204a-204n. The target average bitrate can be the bitrate achieved using general encoding parameters 226 when no other adjustments are made (e.g., no text sharpness, uncertain offsets, no license plate detection, no road sign detection, etc.). In one example, the target average bitrate can be selected by a person with appropriate expertise (e.g., an engineer). In another example, the target average bitrate can be a user-selected input value. In yet another example, the target average bitrate may be constrained by wireless communication device 156. In yet another example, the target average bitrate can be limited to the available communication bandwidth. For example, the target average bitrate can change in real time as communication conditions change. In the example, if available bandwidth decreases (e.g., due to interference, network traffic, hardware failure, etc.), the target average bit rate can be temporarily adjusted to a lower value. The chosen target average bit rate can vary depending on the design criteria of the specific implementation.

[0182] Video encoding module 216 can be configured to perform QP reduction to provide text clarity for license plate ROIs 312a-312b. For example, when a license plate ROI is detected, video encoding module 216 can determine encoding parameter offset 350. Encoding parameter offset 350 can provide an adjustment from general encoding parameter 226. Encoding parameter offset 350 can reduce the value of general encoding parameter 226 to the value of offset parameters 354aa-354bc (e.g., provide a negative offset) to determine text clarity encoding parameter 228. For example, text clarity encoding parameter 228 can be determined by adjusting general encoding parameter 226 downwards. In the example shown, offset parameter 354aa can be a value of -3, offset parameter 354ab can be a value of -10, offset parameter 354ac can be a value of -4, offset parameter 354ba can be a value of -6, offset parameter 354bb can be a value of -6, and offset parameter 354bc can be a value of -7. In some embodiments, each of offset parameters 354aa-354bc can have the same offset value. In some embodiments, each of the offset parameters 354aa-354bc can have the same or different offset values. The QP value can be adjusted to provide individual levels of adjustment, thus providing beneficial results at the granular level. The offset parameters 354aa-354bc shown can provide example offset values ​​for QP reduction. Generally, a smaller QP value (e.g., a larger absolute offset value) can result in better (or clearer) detail in the coding quality at locations 306aa-306mn. The values ​​of the offset parameters 354aa-354bc can be chosen to provide small QP values ​​(but still greater than zero). For example, if the QP value is already very small, then the applied negative offset can be a small number. Specific values ​​of the offset parameters 354aa-354bc can vary depending on the design criteria of a particular implementation.

[0183] Offset parameters 354aa-354bc can be selected to provide a QP reduction, allowing license plate ROIs 312a-312b to be encoded with video quality providing legible text. Video encoding module 216 can also add a QP for general encoding parameters 226 outside the license plate ROI to compensate for the potential increase in the average bit rate of encoded video frames 204a-204n caused by the negative offset of offset parameters 354aa-354bc applied to the license plate ROI. For example, compensation can be made to achieve the same target average bit rate as when no license plate is detected.

[0184] Video encoding module 216 can be configured to provide an adaptive QP offset for encoding uncompressed video frames 202a-202n to generate encoded video frames 204a-204n. In some embodiments, other types of encoding can be implemented. In one example, video encoding module 216 can be configured to perform a force-P-skip. A force-P-skip can use the previous value for encoding blocks 306aa-306mn for the current video frame. A force-P-skip can reduce bit costs by maintaining the exact video data from previous video frames for a specific encoding block in encoding blocks 306aa-306mn. For example, in a video sequence with a large number of cars, an AI-adjusted region of interest (ROI) encoding pipeline 200 can detect license plates only at specific frame intervals and perform license plate tracking using a force-P-skip on the license plate ROI to save bits and maintain high video quality for the ROI. The specific method for selecting the QP value for the encoding block to balance the video quality of the license plate and the total video bitrate can vary depending on the design criteria of the specific implementation.

[0185] refer to Figure 9 A diagram illustrating a portion of an encoded video frame with enhanced text clarity is shown. Encoded video frame portion 380 is shown. Encoded video frame portion 380 may provide an illustrative example of one of the encoded video frames 204a-204n. While the complete encoded video frames 204a-204n can be generated, for illustrative purposes, the illustrated encoded video frame portion 380 may include fewer than the complete encoded video frames. In some embodiments, processor 102 may be configured to crop the encoded video frames 204a-204n to output less than the complete video content of the encoded video frames 204a-204n. In one example, the encoded video frame portion 380 may be a cropped window from video encoded at 1080p30 resolution and frame rate, using H.265 encoding, and having a target bitrate of 2 Mbps. The encoded video frame portion 380 may be encoded using both general encoding parameters 226 (with or without a positive QP offset) and text clarity encoding parameters 228 (e.g., derived from general encoding parameters 226 based on a negative QP offset).

[0186] The encoded video frame portion 380 may include data from... Figure 6A portion of the video data for example video frame 250 is shown in association. The encoded video frame portion 380 may be generated based on preprocessed video frames in the signal PVID. For example, video encoding module 216 may be configured to map the location of license plate ROIs 312a-312b, determined based on scaled-down video frames in the signal DVID, to the original size of the source image in the signal PVID. The encoded video frame portion 380 may include road 252, vehicle 258a with license plate 260a, advertising sign 266, and speed limit road sign 268. Vehicle location bounding box 310a is shown surrounding vehicle 258a, and license plate ROI 312a is shown surrounding license plate 260a. For illustrative purposes, vehicle location bounding box 310a and license plate ROI 312a may be shown. Typically, encoded video frames 204a-204n may be output without displaying indicators and / or visualizations for the bounding boxes (e.g., bounding boxes may be output in debug operation mode).

[0187] The encoded region 382 is shown. The encoded region 382 can be encoded using general encoding parameters 226. The general encoded region 382 can be a portion of the encoded video frames 204a-204n that does not correspond to one of the license plate ROIs 312a-312b. In the example encoded video frame portion 380 shown, the general encoded region 382 can be a portion of the video frame outside the vehicle position bounding box 310a. The general encoded region 382 can also be a portion of the video frame that is within the vehicle position bounding box 310a but outside the license plate ROI 312a.

[0188] An encoded region 384 is shown. The encoded region 384 may be encoded using text clarity encoding parameters 228. For example, the text clarity encoded region 384 may have a QP value equal to the general encoding parameter 226 but offset using offset values ​​354aa-354bc. The text clarity encoded region 384 may be a portion of the encoded video frames 204a-204n corresponding to at least one of the license plate ROIs 312a-312b. In some embodiments, each of the license plate ROIs 312a-312b may be a text clarity encoded region 384 using the same QP value. In some embodiments, each of the license plate ROIs 312a-312b may be an encoded region with different text clarity encoding parameters 228 (e.g., different values ​​for adaptive encoding parameter offset 350, which may be smaller than the general encoding parameter 226).

[0189] License plate character 390 is shown on license plate 260a. The license plate character 390 can be encoded using text-clarity encoding parameters 228. Text-clarity encoded regions 384 can be encoded such that the license plate character 390 is legible in the output encoded video frames 204a-204n. For example, the license plate character 390 in encoded video frames 204a-204n may have the same or slightly lower visual quality compared to the text in the original uncompressed video frames 202a-202n. The license plate character 390 can be readable by someone directly viewing the encoded video frames 204a-204n. For example, the license plate character 390 can be directly viewed and / or read without relying on OCR provided in a separate data stream. In contrast, text on a speed limit road sign 268 encoded using general encoding parameters 226 (e.g., a portion of the general encoded region 382) may not be as clearly output in the encoded video frames 204a-204n as the license plate character 390.

[0190] The speed limit road sign 268 shows sign text 392 and blurred text 394. The speed limit road sign 268 may be in a generic encoded region 382. Because the generic encoded region 382 may be encoded at a lower bit rate using generic encoding parameters 226, the text of the speed limit road sign 268 may suffer a loss of visual sharpness in the encoded video frame portion 380. The amount of visual sharpness of the text on the speed limit road sign 268 may depend on the value of the generic encoding parameter 226, the size of the original text, and / or the sharpness of the original text in the uncompressed video frame. In the example shown, the sign text 392 may be large text representing the number "50". For example, the speed limit road sign 268 may provide the speed limit value in large text. Even when using a lower bit rate from the generic encoding parameter 226, the large text of the sign text 392 may be large and / or sharp enough in the uncompressed video frames 202a-202n to appear clearly in the generic encoded region 382.

[0191] Blurred text 394 can provide an illustrative example of the quality loss of some text in the text on speed limit road signs 268. For example (as with...) Figure 6(As shown in association), the speed limit road sign 268 may include the text "MPH" at the location of the blurred text 394. Typically, road signs display the speed limit value in large text and the speed unit in smaller text. For example, the text size and / or sharpness of "MPH" written on the speed limit road sign 268 may not be large enough and / or sharp enough in the uncompressed video frames 202a-202n, and the quality loss introduced by the general encoding parameter 226 may result in blurred text 394. In the example shown, blurred text 394 may be presented as an irregular shape. In some embodiments, blurred text 394 may appear blocky and / or pixelated in the encoded video frames 204a-204n. By comparison, while general encoding parameter 226 may cause some text (e.g., blurred text 394) to lose quality, text sharpness parameter 228 ensures that license plate character 390 retains a sufficient amount of visual quality to remain visible.

[0192] In some embodiments, to maintain the target average bit rate in the encoded video frames 204a-204n, the QP value for the general encoded region 382 can be increased. For example, the general encoded region 382 may not be encoded using the general encoding parameters 226, but rather using general encoding parameters 226 with a positive offset applied. Applying a positive offset in the general encoded region 382 can compensate for the reduction in QP in the text clarity encoded region 384. However, since a license plate may occupy a small area in the entire video frame, the QP increment used for compensation in the general encoded region 382 may be very small (e.g., compared to the negative offset used for the text clarity encoded region 384). For example, the impact of the QP increment used for the general encoded region 382 on video clarity and / or subjective video quality may be almost imperceptible to the human eye.

[0193] Video encoding module 216 can be configured to determine an adaptive QP offset for each of the license plate ROIs 312a-312b. For example, each of the license plate ROIs 312a-312b can have a unique negative offset for the QP value. Video encoding module 216 can apply different text-clarity encoding parameters 228 for each text-clarity encoded region 384 (e.g., one parameter applied for each license plate detected and within a predetermined distance). Text-clarity encoding parameters 228 can also depend on other detected parameters (e.g., motion blur due to the relative speed of the detected vehicles 258a-258c, the gain level of image sensor 180, the exposure length of image sensor 180, etc.). The QP offset for each of the text-clarity encoded regions 384 can be determined independently to provide a balance between text-clarity and overall video bitrate.

[0194] refer to Figure 10 The diagram illustrates the performance of computer vision operations on an example video frame to detect the location of a road sign. Example video frame 400 is shown. Example video frame 400 may include one of the uncompressed video frames 202a-202n (or a preprocessed version of the uncompressed video frames 202a-202n provided in the PVID signal or a downsized version of the uncompressed video frames 202a-202n provided in the DVID signal). Example video frame 400 may be similar to... Figure 6 Example video frame 250 is shown in association. For example, video frame 400 may represent one of the uncompressed video frames 202a-202n captured before or after example video frame 250.

[0195] Example video frame 400 may include vehicle 80, road 252, lanes 254a-254d, elevated road 256, elevated directional road sign 262, distant sign 264, advertising sign 266, and speed limit road sign 268. In example video frame 400, painted road sign 402 and vehicle 404 are shown. Painted road sign 402 may be a high-occupancy vehicle (HOV) lane symbol. Vehicle 404 may be far from vehicle 80. Because vehicle 404 is far away, example video frame 400 may not include any license plates that are close enough to provide legible text.

[0196] In some embodiments, the object detection CNN 212 may not detect any vehicles within a predetermined distance for text clarity, and the text location detection CNN 214 may not provide any license plate ROI in the absence of vehicle location bounding boxes. In the absence of a license plate ROI, the video encoding module 216 may select general encoding parameters 226 to encode the example video frame 400 at a target average bit rate.

[0197] In some embodiments, the AI-adjusted region of interest (ROI) coding pipeline 200 can provide extended functionality to detect traffic signs, thereby providing textual clarity for traffic sign text in encoded video frames 204a-204n while maintaining a target average bit rate. The operation for providing textual clarity for road signs in encoded video frames 204a-204n can be similar to the operation for providing textual clarity for license plates. First, the traffic sign location bounding box can be detected, and then the text can be located based on the detected sign type to provide the road sign text ROI. Using the determined road sign text ROI, a negative offset to the coding parameters can be applied to provide textual clarity to the coding block corresponding to the road sign text ROI.

[0198] Dashed boxes 410a-410e are shown. Dashed boxes 410a-410e may include bounding boxes of sign locations. Object detection CNN 212 can be configured to perform object detection to detect bounding box locations 410a-410e. In the example, object detection CNN 212 may include a trained AI model 222 configured to determine various types of road signs. For example, the trained AI model 222 implemented by object detection CNN 212 may be configured to detect useful signs but not necessarily perform text recognition (e.g., general OCR). For example, object detection CNN 212 may be able to ignore and / or filter out some types of text (e.g., text on buildings, people holding signs, graffiti, etc.). In one example, a useful sign may be a stop sign, speed limit sign, stop here sign, lane indicator, etc. Typically, a useful sign may be a common sign and / or a symbol on an enumerated set of signs to enable training data. Arbitrary text may not be a useful sign. For example, an arbitrary sign may not have a regular shape and / or color and / or may not provide consistency for training data. The types of road signs detected and / or ignored by the object detection CNN 212 can vary depending on the design criteria of a particular implementation.

[0199] The dashed arrow DSA-DSE is shown. The dashed arrow DSA-DSE can represent a distance measurement performed by the object detection CNN 212. The distance measurement determines the distance to the vehicle 80 (e.g., the location of image sensor 180) and the bounding box 410a-410e of the sign location. Typically, based on the specific classification of road signs, the size of the bounding box for each sign can differ from one another (e.g., an overhead sign may be much larger than a stop sign, resulting in an overhead sign having a larger bounding box size despite being farther away). For example, the object detection CNN 212 can compare the sign location bounding box 410a-410e to road signs of a specific classification to compare the sign to similar reference signs (e.g., comparing the size of a speed limit sign 268 to other speed limit signs). The distance DSA-DSE can be used to filter out signs. Filtering out signs can ignore specific signs that may be too far away to provide easily readable text. For example, the filtering performed on the marker location bounding boxes 410a-410e can be similar to the distance filtering performed on the vehicle location bounding boxes 270a-270c.

[0200] refer to Figure 11 The diagram illustrates flag text detection at the block level of a video frame. An example flag text detection 450 is shown. Flag text detection 450 may include, for example, flag text detection... Figure 10Illustrative examples of video frames 400 described in association. For example, self-vehicle 80, road 252, overhead directional road sign 262, distant sign 264, advertising sign 266, speed limit road sign 268, painted road sign 402, and vehicle 404 are shown. Sign text detection 450 may represent the determination of ROI performed by text location detection CNN 214 in response to signal VLOC and uncompressed video frames 202a-202n. Sign text detection 450 may include vertical lines 302a-302m and horizontal lines 304a-304l forming a grid pattern of coded blocks 306aa-306mn. For example, coded blocks 306aa-306mn may be similar to... Figure 7 The associated coded block is 306aa-306mn.

[0201] Shaded areas 452a-452c are shown. Shaded areas 452a-452c may correspond to the positions of painted road signs 402, overhead directional road signs 262, and speed limit road signs 268. Each of the shaded areas 452a-452c may represent a detected road sign text ROI. Road sign text ROIs 452a-452c may be detected by a text location detection CNN 214 in response to a signal VLOC. The signal TLOC may include road sign text ROIs 452a-452c. Although three road sign text ROIs 452a-452c are shown in the example road sign text detection 450, the number of detected road sign text ROIs may vary based on the number of road signs in uncompressed video frames 202a-202n, the type of sign, and / or the distance from the road sign. Road sign text ROIs 452a-452c may include a total coded block area (e.g., macroblock / CTB area) for road signs to provide text clarity in encoded video frames 204a-204n.

[0202] In Figure 10 In the associated example video frame 400, the object detection CNN 212 may have detected five road sign location bounding boxes 410a-410e. For example, the text location detection CNN 214 may have received five candidates for the road sign text ROI. In one example, the object detection CNN 212 may be configured to detect all text in the video frame as possible candidates for text clarity, and the text location detection CNN 214 may be configured to filter candidates based on size and / or sign type to provide clarity for known sign types. In some embodiments, the object detection CNN 212 may ignore text that includes random / arbitrary text that may be difficult to classify.

[0203] The text location detection CNN 214 can perform filtering based on distance and / or sign type. In the example shown, the distance DSB to the sign location bounding box 410b may be too far (e.g., a distant sign 264 may not have legible text). In the example shown, the advertising sign 266 may include random / arbitrary text, and the sign location bounding box 410d can be filtered out. In some embodiments, the sign location bounding box 410b may be intentionally filtered out to remove annoying and / or distracting content (e.g., ad-blocking features) in the encoded video frames 204a-204n because the sign is classified as an advertisement. After filtering based on distance and / or sign type, the road sign text ROIs 452a-452c can be retained. The encoded blocks 306aa-306mn corresponding to the road sign text ROIs 452a-452c can be presented to the video encoding module 216.

[0204] Video encoding module 216 can be configured to determine an adaptive encoding parameter offset 350 for each of the road sign text ROIs 452a-452c. Similar to the operation performed for license plate ROIs 312a-312b, video encoding module 216 can encode the road sign text ROIs 452a-452c using text clarity encoding parameters 228, and encode the remainder of example video frame 400 using general encoding parameters 226 (or general encoding parameters 226 with a positive offset for compensation). Text clarity encoding parameters 228 ensure that the text within the road sign text ROIs 452a-452c is clear and legible in the encoded video frames 204a-204n, while providing a target average bitrate.

[0205] In Figure 7 In the associated example, only vehicle license plates were detected. Figure 11In the associated example, only road sign text was detected. However, the AI-adjusted region of interest (ROI) coding pipeline 200 can be configured to detect both license plate ROI and road sign ROI in the same video frame. The AI-adjusted ROI coding pipeline 200 can be configured to search various text types, such as license plates and / or road traffic signs with typically predefined shapes / colors. The AI-adjusted ROI coding pipeline 200 can be configured to determine text sharpness coding parameters 228, which are applied to the coding block corresponding to the determined text location, so that the text is sharp in the encoded video output. The AI-adjusted ROI coding pipeline 200 can be implemented to perform the detection operation efficiently, so that high computational resources are not dedicated to performing vision to detect text locations. ROIs can be detected efficiently, thus allowing for adaptive selection of coding to provide sharp text output.

[0206] refer to Figure 12 The diagram illustrates method (or process) 500. Method 500 can provide AI-adjusted ROI encoding to improve the clarity of license plate text in recorded video. Method 500 typically includes steps (or states) 502, 504, 506, 508, decision step (or state) 510, 512, 514, 516, 518, 520, and 522.

[0207] Step 502 can initiate method 500. In step 504, processor 102 can receive pixel data. For example, image sensor 180 can generate a signal VIDEO including pixel data in response to light input LIN captured by capture device 104. Next, in step 506, processor 102 can process the pixel data arranged as video frames. For example, processor 102 can perform various operations on the pixel data arranged as video frames (e.g., perform computer vision operations, calculate depth data, determine white balance, etc.). In one example, video preprocessing pipeline 210 can perform video preprocessing operations on input video frames 202a-202n to generate downsized video frames in signal PVID and / or signal DVID (e.g., uncompressed format). In step 508, processor 102 can perform computer vision operations on the uncompressed video frames. For example, object detection CNN 212 can detect vehicles (or signs) in the uncompressed video frames. Next, method 500 can move to decision step 510.

[0208] In decision step 510, processor 102 can determine whether a vehicle has been detected. In some embodiments, in addition to vehicles, object detection CNN 212 can be configured to detect road signs. If no vehicle (or sign) has been detected, method 500 can return to step 504. If a vehicle (or sign) has been detected, method 500 can move to step 512. In step 512, processor 102 can detect the bounding box locations of one or more detected vehicles (and / or signs). For example, object detection CNN 212 can detect bounding boxes 270a-270b and can generate a signal VLOC. Next, in step 514, processor 102 can perform license plate detection within the bounding box locations. For example, text location detection CNN 214 can search within bounding boxes 270a-270b to detect license plates 260a-260b. Similarly, text can be located within bounding boxes 410a-410e for road signs. In step 516, processor 102 can detect the region of interest for the license plate of a vehicle. For example, the text location detection CNN 214 can determine the coordinates 320a-320d corresponding to ROIs 312a-312b and generate a signal TLOC. Next, method 500 can move to step 518.

[0209] In step 518, processor 102 may determine a first encoding parameter. For example, the first encoding parameter may be a general encoding parameter 226. Next, in step 520, processor 102 may apply an offset to the first encoding parameter to generate a second encoding parameter. For example, offset parameters 354aa-354bc may be applied to general encoding parameter 226 to generate text clarity parameter 228. In step 522, processor 102 may use the first encoding parameter outside ROIs 312a-312b (e.g., general encoding region 382) and the second encoding parameter inside ROIs 312a-312b (e.g., text clarity encoding region 384) to generate encoded video frames 204a-204n. For example, video encoding module 216 may generate a signal TEVID including encoded video frames 204a-204n. Next, method 500 may return to step 504.

[0210] refer to Figure 13The diagram illustrates method (or process) 550. Method 550 can track object positions so that ROI detection can be performed at frame intervals. Method 550 typically includes steps (or states) 552, 554, 556, a decision step (or state) 558, 560, 562, 564, 566, 568, 570, and 572.

[0211] Step 552 can initiate method 550. In step 552, the downsizing module 220 can downsize the input video frames 202a-202n. For example, the video preprocessing pipeline 210 can generate a signal DVID in response to the signal VIDEO. Next, in step 556, the object detection CNN 212 can analyze the downsized video frames using the neural network model 222 to detect vehicle positions. Then, method 550 can move to decision step 558.

[0212] In decision step 558, object detection CNN 212 can determine whether a threshold number of vehicles exist in the scene. For example, when many vehicles are detected in the scene, performing detection at frame intervals may be efficient (e.g., a scene with many vehicles may have many slowly moving vehicles that may not change position rapidly over time). The number of vehicles in the threshold number can vary (e.g., greater than 5). If no vehicles in the threshold number are detected, method 550 can move to step 560. In step 560, text location detection CNN 214 can perform license plate detection on each of the scaled-down video frames in the signal DVID based on the vehicle positions detected in each of the scaled-down video frames (e.g., performed without object tracking). Next, method 550 can return to step 556. In decision step 558, if vehicles in the threshold number are detected, method 550 can move to step 562.

[0213] In step 562, the text location detection CNN 214 can detect license plate ROIs 312a-312b, and the video encoding module 216 can generate an encoded video frame 204i for the current input video frame. For example, the first detection can provide a baseline for tracking objects over time. Next, in step 564, the object detection CNN 212 and / or the text location detection CNN 214 can track the position of the object (e.g., vehicle and / or license plate ROI) in the scaled-down video frame. Then, method 550 can proceed to decision step 566.

[0214] In decision step 566, the object detection CNN 212 and / or text location detection CNN 214 can determine whether a frame interval has been reached. The frame interval can be a predefined number of frames to wait before performing another detection of the vehicle location and / or license plate ROI. If the frame interval has been reached, method 550 can proceed to step 568. In step 568, the object detection CNN 212 and / or text location detection CNN 214 can perform detection of the vehicle and / or license plate ROI at the frame interval. For example, the updated location for the vehicle and / or license plate ROI can be detected at the frame interval. Next, in step 570, the video encoding module 216 can perform video encoding based on the updated license plate ROIs 312a-312b. Then, method 550 can return to step 564.

[0215] In decision step 564, if the frame interval has not yet been reached, method 550 can proceed to step 572. In step 572, video encoding module 216 can perform video encoding using forced P-skip based on the previous position of the license plate ROI. For example, forced P-skip can make the macroblock / CTB results of the previous frame usable for the current video frame. Next, method 550 can return to step 564.

[0216] refer to Figure 14 The diagram illustrates method (or process) 600. Method 600 may apply a positive offset to general coding parameters to provide a consistent bit rate. Method 600 typically includes steps (or states) 602, 604, 606, 608, a decision step (or state) 610, 612, 614, 616, 618, and 620.

[0217] Step 602 can initiate method 600. In step 604, object detection CNN 212 can detect the location (e.g., bounding boxes) of vehicles and / or road signs in the scaled-down video frames. Next, in step 606, text location detection CNN 214 can use neural network model 224 to detect regions of interest for vehicles and / or road signs. In step 608, video encoding module 216 can determine the amount of video frames occupied by regions of interest 312a-312b. Next, method 600 can move to decision step 610.

[0218] In decision step 610, video encoding module 216 can determine whether encoding using text-clarity parameter 228 will cause an increase in the bit rate of encoded video frames 204a-204n exceeding a threshold. For example, the threshold could be the amount of available bandwidth for wireless communication module 156. If the increase in bit rate caused by applying text-clarity parameter 228 to regions of interest 312a-312b does not exceed the threshold, method 600 can move to step 612. In step 612, video encoding module 216 can determine a negative offset from general encoding parameter 226 to generate text-clarity encoding parameter 228. Next, method 600 can move to step 618. In decision step 610, if the increase in bit rate caused by applying text-clarity parameter 228 to regions of interest 312a-312b does indeed exceed the threshold, then method 600 can move to step 614.

[0219] In step 614, video encoding module 216 can determine a negative offset value of 354aa-354bc for generating text-clarity encoding parameters 228. Next, in step 616, video encoding module 216 can determine a positive offset for general encoding parameters 226. A positive offset can be determined to ensure that the bit rate of the encoded video frames 204a-204n remains stable (e.g., within a threshold of available bandwidth). In step 618, video encoding module 216 can generate encoded video frames 204a-204n using general encoding parameters 226 (with a positive offset, if calculated) outside the region of interest and using text-clarity encoding parameters 228 within the region of interest. Next, method 600 can move to step 620. Step 620 can end method 600.

[0220] refer to Figure 15 The diagram illustrates method (or process) 650. Method 650 can filter license plate locations based on distance. Method 650 typically includes steps (or states) 652, 654, 656, 658, decision step (or state) 660, 662, 664, decision step (or state) 666, 668, 670, and 672.

[0221] Step 652 can initiate method 650. In step 654, the object detection CNN 212 can perform the detection of the location of vehicles and / or road signs in the scaled-down video frames. Next, in step 656, the object detection CNN 212 can determine the distance to the detected vehicles and / or signs (e.g., DA-DC and / or DSA-DSE). In step 658, for text clarity, the object detection CNN 212 can compare the distance to a threshold distance. Next, method 650 can move to decision step 660.

[0222] In decision step 660, object detection CNN 212 can determine whether the location of the next object is within a text clarity distance threshold. If the next object is not within the text clarity distance threshold, method 650 can move to step 662. In step 662, object detection CNN 212 can filter out the object from the ROI detection (e.g., the object can be ignored). Next, method 650 can move to decision step 666. In decision step 660, if the location of the next object is within the text clarity threshold, then method 650 can move to step 664. In step 664, text location detection CNN 214 can perform ROI detection for the object (e.g., determine the license plate ROI and / or road sign ROI). Next, method 650 can move to decision step 666.

[0223] In decision step 666, the object detection CNN 212 can determine whether there are any more detected object locations. If there are more detected object locations, method 650 can return to decision step 660. If there are no more detected object locations (e.g., all ROIs have been identified), method 650 can move to step 668. In step 668, video encoding module 216 can calculate negative offset parameters 354aa-354bc based on the distance to each of the remaining ROIs. For example, ROIs at larger distances can use larger negative offsets to compensate for potential loss of sharpness (e.g., smaller text at a certain distance may be less legible after encoding compared to larger text if the same encoding parameters are used). In this example, each of the ROIs can have a different negative offset value. Next, in step 670, video encoding module 216 can encode the uncompressed video frames in the signal PVID using a general encoding parameter 226 outside the ROIs and a different text sharpness encoding parameter 228 for each of the different ROIs. Next, method 650 can move to step 672. Step 672 can end method 650.

[0224] Depend on Figure 1-15The functions executed by the diagram can be implemented using one or more of a conventional general-purpose processor, digital computer, microprocessor, microcontroller, RISC (Reduced Instruction Set Computer) processor, CISC (Complex Instruction Set Computer) processor, SIMD (Single Instruction Multiple Data) processor, signal processor, central processing unit (CPU), arithmetic logic unit (ALU), video digital signal processor (VDSP), and / or similar computing machines programmed according to the teachings of this specification, as will be apparent to those skilled in the art. A skilled programmer can readily prepare appropriate software, firmware, code, routines, instructions, opcodes, microcode, and / or program modules based on the teachings of this disclosure, again as will be apparent to those skilled in the art. The software is typically executed from one or more media by one or more processors in a machine-implemented manner.

[0225] The present invention can also be implemented by preparing an ASIC (Application-Specific Integrated Circuit), a platform ASIC, an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), a CPLD (Complex Programmable Logic Device), a sea-of-gate, an RFIC (Radio Frequency Integrated Circuit), an ASSP (Application-Specific Standard Product), one or more monolithic integrated circuits, one or more chips or dies arranged as flip-chip modules and / or multi-chip modules, or by interconnecting conventional component circuits through a suitable network, as described herein, modifications of which will be apparent to those skilled in the art.

[0226] Therefore, the present invention may also include a computer product, which may be one or more storage media and / or one or more transmission media, including instructions that can be used to program a machine to perform one or more processes or methods according to the present invention. The execution of the instructions contained in the computer product by the machine, and the operation of the surrounding circuitry, can convert input data into one or more files on the storage medium and / or one or more output signals representing physical objects or substances, such as audio and / or visual depictions. The execution of the instructions contained in the computer product by the machine can be performed on data stored on the storage medium and / or user input and / or in combination with values ​​generated by a random number generator implemented by the computer product. The storage medium may include, but is not limited to, any type of disk, including floppy disks, hard disks, magnetic disks, optical disks, CD-ROMs, DVDs, and magneto-optical disks, as well as circuits such as ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), UVPROM (Ultraviolet Erasable Programmable ROM), flash memory, magnetic cards, optical cards, and / or any type of medium suitable for storing electronic instructions.

[0227] The elements of this invention can form part or all of one or more devices, units, components, systems, machines, and / or apparatuses. Devices may include, but are not limited to, servers, workstations, storage array controllers, storage systems, personal computers, laptop computers, notebook computers, handheld computers, cloud servers, personal digital assistants, portable electronic devices, battery-powered devices, set-top boxes, encoders, decoders, transcoders, compressors, decompressors, preprocessors, post-processors, transmitters, receivers, transceivers, cryptographic circuits, cellular phones, digital cameras, positioning and / or navigation systems, medical devices, head-up displays, wireless devices, audio recording, audio storage and / or audio playback devices, video recording, video storage and / or video playback devices, gaming platforms, peripheral devices, and / or multi-chip modules. Those skilled in the art will understand that the elements of this invention can be implemented in other types of devices to meet specific application standards.

[0228] When used herein in conjunction with "is" and verbs, the terms "may" and "usually" are intended to convey the intention that the description is exemplary and is considered broad enough to encompass both the specific examples presented in this disclosure and the alternative examples that may be derived based on this disclosure. As used herein, the terms "may" and "usually" should not be construed as necessarily implying the desirability or possibility of omitting the corresponding element.

[0229] When the notations “a”-“n” are used herein for various components, modules, and / or circuits, a single component, module, and / or circuit or multiple such components, modules, and / or circuits are disclosed, wherein the notation “n” is applied to represent any particular integer. Each different component, module, and / or circuit having an instance (or event) notated as “a”-“n” may indicate that the different component, module, and / or circuit may have a matching number of instances or a different number of instances. An instance designated as “a” may represent the first of multiple instances, and an instance “n” may refer to the last of multiple instances without implying a specific number of instances.

[0230] Although the invention has been specifically shown and described with reference to embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention.

Claims

1. An apparatus comprising: An interface configured to receive pixel data; as well as A processor configured to: (i) process the pixel data arranged as video frames; (ii) Perform computer vision operations on the uncompressed video frames; (iii) detect a bounding box location for the vehicle in response to the computer vision operations; (iv) perform license plate detection within the bounding box location to detect a region of interest for the vehicle's license plate; (v) determine first encoding parameters for generating an encoded video frame based on the uncompressed video frames; and (vi) determine second encoding parameters for the region of interest for the license plate. (vii) using the first encoding parameters outside the region of interest and the second encoding parameters within the region of interest to generate the encoded video frame, wherein, (a) Applying an offset to the first encoding parameter for the region of interest to determine the second encoding parameter; as well as (b) The second encoding parameters provide clarity of the license plate text within the region of interest while maintaining the average bit rate of the encoded video the same as that used to encode a complete video frame using the first encoding parameters.

2. The apparatus according to claim 1, wherein, The processor is configured to implement: (i) a first neural network configured to determine the bounding box position of the vehicle in the video frame; and (ii) a second neural network configured to detect the region of interest of the vehicle's license plate within the bounding box position of the vehicle.

3. The apparatus according to claim 1, wherein, The second encoding parameter enables the encoded video frame to provide the clarity of the text on the license plate without increasing the file size.

4. The apparatus according to claim 1, wherein, (i) the encoded video frame is transmitted via bandwidth-limited wireless communication, and (ii) the average bit rate of the encoded video frame is limited by the bandwidth of the bandwidth-limited wireless communication.

5. The apparatus according to claim 4, wherein, Encoding the video frame using the first encoding parameters instead of the second encoding parameters to achieve the bandwidth of the bandwidth-limited wireless communication results in a compressed video frame with unreadable text for the license plate.

6. The apparatus according to claim 1, wherein, (i) the encoded video frame is generated without performing license plate text recognition, and (ii) the clarity of the text on the license plate enables the license plate number to be easily readable.

7. The apparatus according to claim 6, wherein, The license plate number is not provided along with the encoded video frames.

8. The apparatus according to claim 1, wherein, The offset applied to the first encoding parameter includes a negative value of the quantization parameter.

9. The apparatus according to claim 1, wherein, (i) the uncompressed format of the video frame is YUV format, (ii) the compression format for the encoded video frame is H.265, and (iii) the region of interest of the license plate includes a coding tree block.

10. The apparatus according to claim 1, wherein, (i) the uncompressed format of the video frame is YUV format, (ii) the compression format for the encoded video frame is H.264, and (iii) the region of interest of the license plate includes macroblocks.

11. The apparatus according to claim 1, wherein, The first encoding parameter and the second encoding parameter include block-level video encoding parameters that can be adjusted in real time.

12. The apparatus according to claim 1, wherein, (i) The region of interest includes a total coding block area corresponding to the position of the license plate in the video frame, (ii) the total coding block area includes a plurality of squares, and (iii) the region of interest includes a rectangular shape having a pixel height and pixel width corresponding to the position of the license plate in the video frame.

13. The apparatus according to claim 1, wherein, The bounding box position and the region of interest of the license plate are detected in each video frame of the video frame.

14. The apparatus according to claim 1, wherein, (i) the bounding box position and the region of interest of the license plate are detected in the video frame at predetermined frame intervals, and (ii) the processor is further configured to perform predictive tracking of the bounding box position and the region of interest of the license plate for a subset of the video frames, wherein the subset of the video frames is captured between the predetermined frame intervals.

15. The apparatus according to claim 1, wherein, (i) A positive offset is applied to the first encoding parameter to compensate for the offset applied to the second encoding parameter, (ii) the positive offset is selected to compensate for an increase in the average bit rate caused by the offset applied to the second encoding parameter, and (iii) the positive offset of the first encoding parameter provides an imperceptible reduction in video quality to portions of the video frame outside the region of interest of the license plate.

16. The apparatus according to claim 1, wherein, When selecting the offset for the second encoding parameter, the clarity of the text on the license plate is given priority, while the subjective video quality of the encoded video frame is disregarded.

17. The apparatus according to claim 1, wherein, (i) the video frame includes a 1080p resolution, and (ii) the offset for the second encoding parameter is provided to achieve the clarity of the text of the license plate without increasing the resolution of the video frame.

18. The apparatus according to claim 1, wherein, The processor is further configured to: (i) detect the location of a road sign bounding box in response to the computer vision operation; (ii) perform road sign text detection within the road sign bounding box location to detect a region of interest (ROI) of the road sign text; and (iii) use the second encoding parameter on the ROI to provide clarity of the road sign text.

19. The apparatus according to claim 1, wherein, (a) The processor is further configured to: (i) determine the distance to the vehicle based on the size of the bounding box position; and (ii) compare the distance to the vehicle with a predetermined distance. (iii) If the distance is greater than the predetermined distance, the license plate location is filtered out from the license plate detection performed within the bounding box location, and (b) the text of the license plate that is beyond the predetermined distance is not easily readable in the video frame.

20. The apparatus according to claim 1, wherein, (a) The processor is further configured to: (i) determine the relative speed of the vehicle with respect to the device; and (ii) receive image sensor parameters for capturing the video frames; (iii) determining the distortion level of the vehicle based on the relative speed and the image sensor parameters, and (b) the offset being further determined in response to the distortion level of the vehicle.

Images