Apparatus and computer-implemented method for intermediate frame image processing
Through the intermediate frame image processing scheme, image data processing is performed in a multi-sensor environment using an image frame multiplexer and a processor, which solves the problem of delay and cost of switching between image sensors in the prior art, and realizes efficient image processing Throughput and low-cost image processing system.
Patent Information
- Application Number
- CN202011282317.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-11-14
- Filing Date
- 2020-11-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-16
AI Technical Summary
When existing imaging devices perform image processing in multi-sensor environments, there are problems of increased system delay, equipment complexity and high cost, especially in dynamic switching and synchronization operations between image sensors.
The intermediate frame image processing scheme is adopted, and image data processing and transmission is performed using the intermediate frame image size through the image frame multiplexer and processor configuration, so as to achieve seamless switching between image sensors and reduce device complexity and cost.
This improves image processing throughput and reaction time, reduces equipment costs, and realizes efficient image processing in multi-sensor environments.
Smart Images

Figure CN112801850B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate generally to image processing, and in particular, to apparatus and computer-implemented methods for performing image processing in a multi-sensor environment utilizing a common image data format (eg, intermediate frame image size) among all image sensors. Background Art
[0002] In a conventional imaging device including two or more image sensors, the images output by the image sensors need to be combined into one output for processing. Similarly, signals need to be synchronized between two or more sensors to maintain system efficiency and efficacy. To perform such operations, conventional imaging devices traditionally utilize programmable logic devices, such as complex programmable logic devices ("CPLDs") or field programmable gate arrays ("FPGAs"). Such computing devices add system delays to one or more frames and increase the complexity, cost, and size of the device. In addition, conventional imaging devices traditionally utilize dynamic switching of image sensors. Conventional dynamic switching between two or more sensors traditionally requires operating system and / or driver support, and traditionally requires reinitialization of each image sensor when switching between two or more sensors, further reducing the system's snappiness. The applicant has discovered problems with the current implementation of conventional image processing in a multi-sensor environment with two or more image sensors. Through the efforts, wisdom, and innovations applied, the applicant has solved many of these recognized problems by developing solutions embodied in the present disclosure, which will be described in detail below. Summary of the invention
[0003] In general, embodiments of the present disclosure provided herein include systems, methods, apparatus, and computer program products for intermediate frame image processing. Other systems, apparatus, methods, computer-readable media, and features will be or will become apparent to those skilled in the art upon review of the following drawings and detailed description. It is intended that all such additional systems, apparatus, methods, computer-readable media, and features included in this specification be within the scope of the present disclosure and protected by the following claims.
[0004] According to one aspect of the present disclosure, an intermediate frame image processing device includes a first image sensor configured to output a first intermediate frame image data object based on a first full frame image size and an intermediate frame image size. The exemplary intermediate frame image processing device also includes a second image sensor configured to output a second intermediate frame image data object based on a second full frame image size and an intermediate frame image size. The exemplary intermediate frame image processing device also includes an image frame multiplexer that performs data communication with the first image sensor and the second image sensor and is configured to receive the first intermediate frame image data object from the first image sensor and receive the second intermediate frame image data object from the second image sensor. The image frame multiplexer is further configured to output an intermediate frame image data object selected from the first intermediate frame image data object or the second intermediate frame image data object. The exemplary intermediate frame image processing device also includes a processor capable of communicating with the image frame multiplexer, the processor being configured to receive the selected intermediate frame image data object from the image frame multiplexer and process the selected intermediate frame image data object.
[0005] Additionally or alternatively, in at least some exemplary embodiments of the apparatus, at least a first image sensor is configured based on an intermediate frame image size and outputs an intermediate frame image data object, the first image sensor being configured to generate a first intermediate frame image data object from a first full frame image data object by performing one or more of the following operations: (1) cropping the first full frame image data object to include in the first intermediate frame image data object: a cropped image data portion of the first full frame image data object, the cropped image data portion being identified based on the first full frame image size and the intermediate frame image size, and (2) padding the first intermediate frame image data object based on image data padding, the image data padding being identified based on the first full frame image size and the intermediate frame image size, and transmitting the first intermediate frame image data object to an image frame multiplexer.
[0006] Additionally or alternatively, in at least some exemplary embodiments of the apparatus, the first full frame image size differs from the second full frame image size by at least one or more data rows and / or one or more data columns.
[0007] Additionally or alternatively, in at least some exemplary embodiments of the apparatus, the first full frame image size is equal to the second full frame image size by having the same number of data rows and the same number of data columns.
[0008] Additionally or alternatively, in at least some exemplary embodiments of the device, the processor is configured to process the selected intermediate frame image data object to determine useful data loss in the selected intermediate frame image data object. Additionally or alternatively, in some exemplary embodiments of the device, the processor is further configured to identify a selected image sensor of the first image sensor or the second image sensor based on the selected intermediate frame image data object. Additionally or alternatively, in some embodiments of the device, the processor is further configured to transmit a full frame request signal to the selected image sensor. Additionally or alternatively, in some embodiments of the device, the processor is further configured to receive a selected full frame data object from the selected image sensor, the full frame image data object including more data than the selected intermediate frame image data. Additionally or alternatively, in some embodiments of the device, the processor is further configured to process the selected full frame image data object.
[0009] Additionally or alternatively, in at least some exemplary embodiments of the device, the processor is further configured to configure at least one of the first image sensor and the second image sensor so that the first image sensor outputs a first intermediate frame image data object based on the intermediate frame image size, and the second image sensor outputs a second intermediate frame image data object based on the intermediate frame image size. Additionally or alternatively, in at least some exemplary embodiments of the device, to configure at least one of the first image sensor and the second image sensor, the device is configured to: identify a first full frame image size associated with the first image sensor and a second full frame image size associated with the second image sensor; determine a maximum image size based on the first full frame image size and the second full frame image size; determine a minimum image size based on the first full frame image size and the second full frame image size; generate an intermediate frame image size based on the maximum image size and the minimum image size; and configure at least one of the first image sensor and the second image sensor based on the intermediate frame image size.
[0010] Additionally or alternatively, in at least some exemplary embodiments of the apparatus, the image frame multiplexer is configured to receive a frame selection signal from the processor, the frame selection signal indicating which of the first intermediate frame image data object or the second intermediate frame image data object is to be selected. Additionally or alternatively, in at least some such exemplary embodiments of the apparatus, the processor is further configured to generate the frame selection signal using a random frame selection algorithm or a sequential frame selection algorithm; and transmit the frame selection signal to the image frame multiplexer. Additionally or alternatively, in at least some exemplary embodiments of the apparatus, the processor is further configured to generate the frame selection signal by processing the received selected intermediate frame image data object; and transmit the frame selection signal to the image frame multiplexer.
[0011] According to yet another aspect of the present disclosure, a computer-implemented method for intermediate frame image processing is provided. In one or more embodiments, the computer-implemented method is implemented via computing hardware, circuits, etc. as described herein. In at least one exemplary embodiment of the computer-implemented method, the exemplary computer-implemented method includes outputting a first intermediate frame image data object from a first image sensor and a second intermediate frame image data object from a second image sensor to an image frame multiplexer, the first intermediate frame image data object and the second intermediate frame image data object being associated with an intermediate frame image size. The computer-implemented method also includes selecting, by the image frame multiplexer, a selected intermediate frame image data object including the first intermediate frame image data object or the second intermediate frame image data object. The computer-implemented method also includes receiving, by a processor, the selected intermediate frame image data object from the image frame multiplexer. The computer-implemented method also includes processing, by the processor, the selected intermediate frame image data object.
[0012] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, outputting a first intermediate frame image data object from the first image sensor to the image frame multiplexer includes: capturing, by the first image sensor, a first full frame image data object associated with a first full frame image size; identifying, by the first image sensor, the intermediate frame image size; identifying, by the first image sensor, a cropped image data portion of the first full frame image data object based on the first full frame image size and the intermediate frame image size; identifying, by the first image sensor, a first image data padding based on the first full frame image size and the intermediate frame image size; generating, by the first image sensor, a first intermediate frame image data object based on the cropped image data portion of the first full frame image data object and the first image data padding; and transmitting, by the first image sensor, the first intermediate frame image data object to the image frame multiplexer.
[0013] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method,
[0014] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, the first image sensor is associated with a first full-frame image size and the second image sensor is associated with a second full-frame image size, wherein the first full-frame image size differs from the second full-frame image size by at least one or more data rows and / or one or more data columns.
[0015] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, the first image sensor is associated with a first full frame image size and the second image sensor is associated with a second full frame image size, wherein the first full frame image size and the second full frame image size are equal by having the same number of data rows and the same number of data columns.
[0016] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, processing by the processor a selected intermediate frame image data object includes: processing by the processor the selected intermediate frame image data object to determine that useful data in the selected intermediate frame image data object is lost, and the computer-implemented method further includes: identifying by the processor a selected image sensor from the first image sensor or the second image sensor based on the selected intermediate frame image data object; transmitting by the processor a full frame request signal to the selected image sensor; receiving by the processor a selected full frame data object from the selected image sensor, the full frame image data object including more data than the selected intermediate frame image data object; and processing by the processor the selected full frame image data object.
[0017] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, the computer-implemented method further comprises configuring, by the processor, the first image sensor to output a first intermediate frame image data object based on the intermediate frame image size. Additionally or alternatively, in at least some such exemplary embodiments of the computer-implemented method, configuring the first image sensor to output a first intermediate frame image data object based on the intermediate frame image size comprises: identifying, by the processor, at least a first full frame image size associated with the first image sensor and a second full frame image size associated with the second image sensor; determining, by the processor, a maximum image size based on the first full frame size and the second full frame image size; determining, by the processor, a minimum image size based on the first full frame image size and the second full frame image size; generating, by the processor, an intermediate frame image size based on the maximum image size and the minimum image size; and configuring, by the processor, the first image sensor to output the first intermediate frame image data object based on the intermediate frame image size.
[0018] Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, selecting the selected intermediate frame image data object by the image frame multiplexer includes: receiving, by the image frame multiplexer, a frame selection signal from a processor, the frame selection signal indicating which intermediate frame image data object of a plurality of intermediate frame image data objects to select; and selecting, by the image frame multiplexer, the selected intermediate frame image data object from the first intermediate frame image data object and the second intermediate frame image data object based on the frame selection signal. Additionally or alternatively, in at least some such exemplary embodiments of the computer-implemented method, the computer-implemented method further includes generating, by the processor, the frame selection signal using a random frame selection algorithm or a sequential frame selection algorithm; and transmitting, by the processor, the frame selection signal to the image frame multiplexer. Additionally or alternatively, in at least some exemplary embodiments of the computer-implemented method, the computer-implemented method further includes: generating, by the processor, the frame selection signal by processing the received selected intermediate frame image data object; and transmitting, by the processor, the frame selection signal to the image frame multiplexer.
[0019] According to another aspect of the present disclosure, a computer program product for intermediate frame image processing is provided. At least one exemplary computer program product includes a non-transitory computer-readable storage medium having computer program instructions stored thereon. The computer program instructions executed by at least one processor are configured to perform one or more of the identified computer-implemented methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Having thus generally described embodiments of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:
[0021] Figure 1 An intermediate frame image processing apparatus according to at least some exemplary embodiments of the present disclosure is shown;
[0022] Figure 2 Components and corresponding data flows of an exemplary intermediate frame image processing apparatus according to at least some exemplary embodiments of the present disclosure are shown;
[0023] Figure 3 illustrates an exemplary visualization of the functionality of an image frame multiplexer for selecting intermediate image data objects in a dual sensor environment in accordance with at least some exemplary embodiments of the present disclosure;
[0024] Figure 4 shows an exemplary intermediate frame image data object in a multi-sensor environment according to at least some exemplary embodiments of the present disclosure;
[0025] Figure 5shows an exemplary flow chart including exemplary operations for intermediate frame image processing according to at least some exemplary embodiments of the present disclosure;
[0026] Figure 6 An exemplary flow chart according to at least some exemplary embodiments of the present disclosure is shown, the exemplary flow chart including additional operations for intermediate frame image processing, specifically including additional operations for corresponding full frame image data object processing;
[0027] Figure 7 An exemplary flow chart according to at least some exemplary embodiments of the present disclosure is shown, the exemplary flow chart including additional operations for intermediate frame image processing, specifically including additional operations for configuring one or more image sensors for outputting intermediate frame image data objects;
[0028] Figure 8 An exemplary flow chart is shown, according to at least some exemplary embodiments of the present disclosure, including operations for generating an intermediate frame image data object, such as would be performed by an image sensor;
[0029] Fig. 9 An exemplary flow chart is shown, including exemplary specific implementations of operations for generating an intermediate frame image data object, such as to be performed by an image sensor, in accordance with at least some exemplary embodiments of the present disclosure; and
[0030] Fig.10 An exemplary flowchart according to at least some exemplary embodiments of the present disclosure is shown, which includes additional operations for generating intermediate frame image data objects, such as to be performed by an image sensor, and specifically includes additional operations for filling and / or cropping full frame image data objects to generate corresponding intermediate frame image data objects. DETAILED DESCRIPTION
[0031] Embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some but not all embodiments of the present disclosure are shown. Indeed, embodiments of the present disclosure can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, but rather, these embodiments are provided so that the present disclosure will satisfy applicable legal requirements. Throughout the text, like reference numerals refer to like elements.
[0032] Overview
[0033] Embodiments of the present disclosure solve various computational and cost inefficiencies of conventional imaging devices. Specifically, embodiments of the present disclosure utilize exemplary intermediate frame schemes to provide various advantages over conventional imaging devices and corresponding conventional schemes. In this regard, in some examples, embodiments of the present disclosure improve overall image processing throughput (e.g., frame rate for processing captured data) and reaction time (e.g., time between triggering capture and indication of successful capture and processing of captured images, such as time between triggering capture and receiving indication of successful decoding of barcodes). In order to improve the reaction time of the device, in at least some embodiments described herein, the time delay for switching between different image sensors is reduced. In this regard, some embodiments of the present disclosure implement seamless switching between different image sensor channels. In addition, some embodiments of the present disclosure do not utilize programmable logic devices, and such exemplary embodiments implement smaller form factors that can be included in smaller size devices. Similarly, in this regard, some such exemplary embodiments include components that are less expensive than programmable logic device modules, thereby enabling such embodiments to be manufactured with improved cost efficiency compared to conventional devices.
[0034] In some examples, the intermediate frame scheme utilizes a specific intermediate frame image format on all image sensors of a particular device. In this regard, the intermediate frame image format embodies an intermediate frame image size that defines a specific number of rows and columns within an intermediate frame image data object (e.g., an "intermediate frame row count" and an "intermediate frame column count"). The intermediate frame size can be determined based on a plurality of associated image sensors. Each of the plurality of image sensors can be associated with any resolution, pixel architecture, and data rate. In this regard, each image sensor can be configured to capture a full frame image data object having a specific image size (e.g., a full frame image size), and then generate an intermediate frame image data object from the full frame image data object for output. Therefore, the intermediate frame size represents the same image size of the image data object output by each of the plurality of image sensors. In some embodiments, the image sensor is specifically configured based on the intermediate frame image size, which utilizes the resolution and bandwidth associated with the plurality of image sensors to maximize the advantages of the present invention.
[0035] In this regard, exemplary embodiments disclosed herein include multiple image sensors that each generate an intermediate frame image data object, each image sensor having the same size. The intermediate frame image data objects are each associated with the same intermediate frame image size, regardless of the resolution, pixel architecture, and / or data rate of each image sensor. In this way, the intermediate frame scheme enables seamless switching between multiple image sensors using smaller and more cost-effective components. In addition, in some examples, the intermediate frame scheme advantageously improves device throughput and reaction time.
[0036] definition
[0037] In some embodiments, some operations in the operations herein may be modified or further amplified. In addition, in some embodiments, additional optional operations may also be included. Modifications, amplifications or additions to the operations herein may be performed in any order and in any combination.
[0038] Those skilled in the art to which the present invention belongs will think of many modifications and other embodiments of the present disclosure set forth herein after benefiting from the teachings presented in the foregoing description and the associated drawings. Therefore, it should be understood that the present invention is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. In addition, although the above description and the associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, it is also conceivable to combine elements and / or functions that are different from those explicitly described herein, as may be shown in some of the appended claims. Although specific terms are employed herein, they are used only in a general and descriptive sense, and not for the purpose of limitation.
[0039] As used herein, the terms "data object," "data," "content," "information," and similar terms may be used interchangeably to refer to data capable of being transmitted, received, and / or stored in accordance with embodiments of the present invention. Additionally, a data object may refer to a value captured, generated, or derived by the embodiments described herein. Thus, the use of any such terms should not be construed as limiting the spirit and scope of the embodiments of the present disclosure.
[0040] The term "processor" refers to computing hardware, firmware, and / or software for executing one or more instructions representing computer-implemented actions. In some embodiments, the processor includes one or more memory components for storing instructions and / or data objects for manipulation. In addition or alternatively, in some embodiments, the processor is configured to communicate with one or more memory devices for retrieving instructions and / or storing and retrieving data objects for manipulation. Non-limiting examples of processors include specially configured processing circuits, microprocessors, central processing units ("CPUs"), digital signal processors ("DSPs"), and / or combinations thereof.
[0041] The term "image frame multiplexer" refers to computing hardware, software, firmware, and / or a combination thereof for selecting between various image data objects received as input signals, wherein the selected image data object is output by the image frame multiplexer. In some embodiments, the image frame multiplexer is controlled by a control signal referred to as a "frame selection signal", which indicates which input data object is to be selected for output. In some embodiments, the frame selection signal is received from an associated processor (e.g., a processor to which the selected image frame data object is to be output). In other embodiments, the image frame multiplexer receives a frame selection signal from a computing device and / or computing hardware configured to receive an alternative processor of the output image data object. In at least some embodiments, the image frame multiplexer is embodied by switching hardware that connects multiple input lines to a single output line, and the image frame multiplexer receives a signal for selecting one of the input lines for output via an output line. In some embodiments, the image frame multiplexer is fully embodied in software for controlling a flow from a selected image sensor to a processor for processing.
[0042] The term "image size" refers to a data value representing a dimensional attribute of an image data object. In the case of a two-dimensional image, the image size represents a specific width-height resolution of the image data object defined by a specific number of columns and rows. Each row-column pair represents a separate data value in the corresponding image data object. For example, a "1920×1200" image size refers to a 1920×1200 resolution with 1920 columns of data and 1200 rows of data. It should be understood that in some embodiments, one or more image sizes are associated with a specific aspect ratio, and the specific aspect ratio of two image sizes may be the same or different.
[0043] The term "full-frame image data object" refers to image data captured by an image sensor. A full-frame image data object is associated with a "full-frame image size," which refers to the image size of the full-frame image data object. In some embodiments, the image sensor is configured to capture a full-frame image data object of a particular full-frame image size using a single mode. In other embodiments, the image sensor is configured to enable selection from a plurality of modes, each mode being associated with adjusting the full-frame image size such that a full-frame image data object captured by the image sensor has an adjusted full-frame image size for the set mode.
[0044] The term "intermediate frame image data object" refers to an image data object generated based on a captured full frame image data object and an intermediate frame image size. The intermediate frame image data object is generated from the full frame image data object based on the difference between the full frame image size and the intermediate frame image size of the full frame image data object. In some embodiments, the intermediate frame image data object includes a cropped portion of the corresponding full frame image data object to reduce the number of rows and / or columns from the full frame image size and the intermediate frame image size, and / or includes padding data to increase the number of rows from the full frame image size to the intermediate frame image size.
[0045] The term "image sensor" refers to computing hardware for capturing image data embodied by a corresponding full-frame image data object. The image sensor is configured to capture image data based on a specific full-frame image size (e.g., a pixel count associated with the image sensor). In addition, in some embodiments, the term "image sensor" further refers to supporting circuitry / hardware, firmware, and / or software for configuring data captured and / or output by the image sensor. For example, the term "image sensor" may further refer to supporting circuitry for generating intermediate frame image data objects from a captured full-frame image data object and / or subsequently outputting intermediate frame image data objects. Non-limiting examples of image sensors include the AR0234 global shutter monochrome sensor (1920×1200 active pixel array resolution, 3μm pixels) and the AR0144 global shutter monochrome sensor (1280×800 active pixel array resolution, 3μm pixels), both of which are manufactured by ON Semiconductor, Inc., headquartered in Phoenix, Arizona. When used herein with reference to an image sensor, the term "resolution" refers to the active pixel array resolution of the corresponding image sensor. When used herein with reference to one or more image data objects, the term "resolution" refers to the number of data rows and the number of data columns represented by the image data object. It should be understood that the resolution of one or more image data objects may depend on the resolution of the corresponding image sensor.
[0046] Exemplary Apparatus and Data Flow
[0047] Figure 1 An exemplary device according to at least one exemplary embodiment of the present disclosure is shown. Specifically, Figure 1 An exemplary intermediate frame imaging device according to at least one exemplary embodiment of the present disclosure is shown, specifically a dual-sensor intermediate frame imaging device. As shown, the intermediate frame imaging device 100 includes an image frame multiplexer 104 and a processor 106. In addition, the intermediate frame imaging device 100 includes a plurality of image sensors 102A to 102N (collectively referred to as "image sensors 102"). It should be understood that in various embodiments, the intermediate frame imaging device 100 includes any number of image sensors 102. For example, in some embodiments, the intermediate frame imaging device 100 includes two image sensors (e.g., image sensor 102A and image sensor 102B) for capturing image data for processing. Alternatively, in other embodiments, the intermediate frame imaging device 100 embodies a three-sensor imaging device having three image sensors (e.g., image sensor 102A, image sensor 102B, and an additional image sensor not shown). In other embodiments, the intermediate frame imaging device 100 includes four or more image sensors.
[0048] Each of the image sensors 102 embodies a computing device configured to capture image data from the environment of the intermediate frame imaging apparatus 100. In addition, each of the image sensors 102 is configured to capture image data of a specific resolution / size. In this regard, each of the image sensors 102 is configured to capture a full-frame image data object having a corresponding full-frame image size. It should be understood that each of the image sensors 102 may be coupled to an optical component (including, but not limited to, one or more lenses, windows, reflective components, refractive components, etc.) to enable capturing image data associated with a specific field. In addition, it should be understood that each of the image sensors 102 may be embodied by computing hardware configured to capture such image data using any known method, including, but not limited to, a charge coupled device ("CCD") and a complementary metal oxide semiconductor ("CMOS") sensor device, their equivalents, or any combination thereof.
[0049] In some embodiments, each of the image sensors 102 is configured to generate an intermediate frame image data object for output. In this regard, in some embodiments, each of the image sensors 102 is configured to store, retrieve, and / or otherwise access an intermediate frame image size for use in generating the intermediate frame image data object. Each of the image sensors 102 may be configured by another component of the intermediate frame imaging device 100 (e.g., the processor 106) or by external computing hardware, software, and / or firmware (not shown). In some embodiments, a subset of the image sensors 102 is configured to generate and / or output intermediate frame image data objects, for example, where the image sensor 102A is configured to output a full frame image data object of a particular full frame image size, and the remaining image sensors 102 are configured to output intermediate frame image data objects of an intermediate frame image size that matches the full frame image size.
[0050] In some embodiments, each of the image sensors 102 is configured to capture full-frame image data objects associated with a variety of different full-frame image sizes. In other embodiments, one or more image sensors are configured to capture full-frame image data objects of the same full-frame image size. In this regard, each of the image sensors 102 may be associated with capturing image data from different depths of a particular environment (e.g., near field and far field), different portions of the same field, or different environments representing two different fields.
[0051] In yet other embodiments, each image sensor 102 is configured to capture an intermediate frame image data object. In this regard, the image sensor may be configured to capture a particular region of interest based on the intermediate frame image size and output the captured data alone or together with additional data rows and / or additional data columns, such as in the form of image fill data. In this regard, each configured image sensor in the image sensors 102 is configured to output an intermediate frame image data object without requiring a post-processing step for manipulating the captured image data, such as such that manipulation by cropping and / or padding the captured full frame image data object is not required.
[0052] In some embodiments, one or more of the image sensors 102 are configured to enable activation of various modes, each mode being associated with capturing and / or outputting a different image data object. In some embodiments, one or more of the image sensors 102 are configured to enable activation of different modes for capturing full-frame image data objects of different full-frame image sizes. In addition or alternatively, in some embodiments, one or more of the image sensors 102 are configured to enable switching between various output modes. For example, in some embodiments, one or more of the image sensors 102 are configured to enable activation of a first mode for outputting a captured full-frame image data object and activation of a second mode for outputting an intermediate frame image data object based on the captured full-frame image data object.
[0053] The image frame multiplexer 104 embodies computing hardware for receiving multiple inputs, and based on a control signal, selects one of the multiple input channels for output. For example, in some embodiments, the image frame multiplexer 104 includes a circuit for receiving a data object output by the image sensor 102 and then outputting a data object selected from various inputs. As shown, the image frame multiplexer 104 is configured to receive data output by each image sensor in the image sensor 102 as various inputs. In an exemplary case, the image frame multiplexer 104 receives an intermediate frame image data object from each image sensor in the image sensor 102, wherein each intermediate frame image data object includes at least a portion of a corresponding full frame image data object captured by a corresponding image sensor in the image sensor 102. The image frame multiplexer 104 is configured to output one of the received image data objects (e.g., one of the intermediate frame image data objects) based on a frame selection signal received from another computing device, hardware, etc. In at least one exemplary embodiment, the frame selection multiplexer is configured to receive a frame selection signal from the processor 106, as described herein.
[0054] The processor 106 embodies computing hardware, a computing device, or a combination of computing devices configured to perform various specifically programmed operations. In some embodiments, the processor 106 is embodied by a processing circuit configured to perform the operations described herein. In other embodiments, the processor 106 is embodied by a microprocessor. In other embodiments, the processor 106 is embodied by a CPU. The processor 106 may include or be otherwise associated with one or more memory devices storing instructions representing various operations described herein, and / or for storing and / or receiving data objects and / or other data values for one or more operations. The processor 106 may be configured to receive data (e.g., output image frame data objects) from the image frame multiplexer 104, and / or transmit data (e.g., frame selection signals) to the image frame multiplexer 104. In addition or alternatively, in some embodiments, the processor 106 is configured to communicate with some or all of the image sensors 102, for example, to set one or more modes based on a particular intermediate frame image size and / or to configure the image sensor 102 to generate and output intermediate frame image data objects.
[0055] In some embodiments, an image sensor in the image sensor 102 is selected without the image frame multiplexer 104. In some such embodiments, for example, the processor 106 communicates directly with each of the image sensors 102. In this regard, the output lines from each of the image sensors 102 are connected together, and only one of the image sensors 102 is activated at a given time (e.g., a frame of time). When a particular image sensor is selected, the remaining image sensors in the image sensor 102 may be disabled or otherwise set to a high impedance state. In this way, an embodiment may receive a particular intermediate frame from a selected sensor in the image sensor 102 without utilizing an image frame multiplexer, such as the image frame multiplexer 104. In another embodiment, multiple image sensors are connected to multiple input ports of the processor, so that multiple image sensors can be implemented for image capture, but only data to be used in processing is selected from a single input port associated with the selected image sensor. For example, in some such embodiments, the output line from the image sensor 102A may be directly connected to a first port of the processor 106, and the output line from the image sensor 102B may be directly connected to a second port of the processor 106. Image sensors 102A and 102B may each be configured to capture intermediate frame image data objects and transmit such intermediate frame image data objects to processor 106 via respective output lines. Furthermore, in some such embodiments, processor 106 is configured to determine the image sensor selected from between image sensor 102A and image sensor 102B and receive the selected intermediate frame image data object via an appropriate input port corresponding to the selected image sensor. Processor 106 may then process the received intermediate frame image data object.
[0056] Figure 2 Components and corresponding data flows of an intermediate frame image processing apparatus according to at least some exemplary embodiments of the present disclosure are shown. As shown in the figure, Figure 2Various configured hardware components are depicted (shown in solid lines), including image sensors 202A and 202B configured to generate, transmit and / or receive various data objects (shown in dashed or "dotted lines"), image frame multiplexer 206, and processor 212, wherein the various data objects include intermediate frame image data object 204A, intermediate frame image data object 204B, selected image data object 210, and frame selection signal 208. Specifically, as shown, image sensors 202A and 202B are capable of communicating with image frame multiplexer 206, such as such that each image sensor is configured to provide intermediate frames (e.g., intermediate frame image data object 204A and intermediate frame image data object 204B) as input to image frame multiplexer 206. The image frame multiplexer 206 can also be in communication with the processor 212, such that, for example, the image frame multiplexer 206 is configured to receive a frame selection signal 208 from the processor 212 and output corresponding output data objects (e.g., the selected image data object 210) from the various inputs. The specific hardware components 202A, 202B, 206, and 212 shown may each embody components of a specific imaging device (such as the intermediate frame imaging device 100). In this regard, the above description with respect to Figure 1 The computing hardware, circuitry, software, and / or firmware depicted in the illustrated and described intermediate-frame imaging apparatus 100 may embody similarly named components.
[0057] Figure 2 An exemplary dual sensor environment including image sensor 202A and image sensor 202B is shown. In one such exemplary embodiment, image sensor 202A embodies a 2.45 megapixel ("MP") sensor, and image sensor 202B embodies a 1.0MP sensor. In an exemplary case, a dual sensor environment is utilized in a device embodying a near-far imaging engine, for example, where image sensor 102A is used to capture a wide image of a near field to serve as a near-field image sensor, and image sensor 202B is used to capture a narrow image of a far field to serve as a far-field image sensor. In an exemplary case, image sensor 202A and image sensor 202B can each capture a full-frame image data object associated with a different full-frame image size. In some such embodiments, the full-frame image sizes each match a specific design aspect ratio. In other embodiments, the full-frame image sizes are each associated with a different aspect ratio.
[0058] Image sensor 202A and image sensor 202B are each configured to generate and output intermediate frame image data objects from the captured full frame image data objects. Specifically, as shown, image sensor 202A is configured to output intermediate frame image data objects 204A, and image sensor 202B is configured to output intermediate frame image data objects 204B. The generated intermediate frame image data objects are each associated with a specific intermediate frame image size that is the same between image sensors 202A and 202B. In this regard, image sensor 202A and image sensor 202B have each been previously configured, for example by processor 212 and / or external computing hardware, based on the intermediate frame image size to output intermediate frame image data objects.
[0059] As shown, image sensor 202A is configured to capture a first full-frame image data object associated with a first full-frame image size represented by (R1, C1), where R1 represents the number of rows in the first captured full-frame image data object (referred to as the "full-frame row count"), and C1 represents the number of columns in the first captured full-frame image data object (referred to as the "full-frame column count"). Image sensor 202B is configured to capture a second full-frame image data object associated with a second full-frame image size represented by (R2, C2), where R2 represents the full-frame row count of the second captured full-frame image data object, and C2 represents the full-frame column count of the second captured full-frame image data object. In this regard, the particular image sizes each represent a resolution P, where P=R*C. In this regard, continuing the above exemplary case, the first full-frame image size represents a first resolution P1, where P1=R1*C1, and the second full-frame image size represents a second resolution P2, where P2=R2*C2. Furthermore, where image sensor 202A embodies a 2.45MP sensor and image sensor 202B embodies a 1.0MP sensor, it should be appreciated that the first full-frame image size represents a first resolution that is greater than a second resolution represented by the second full-frame image size, such that P1 > P2. In some such exemplary cases, the first full-frame image size and the second full-frame image size satisfy the properties R1 >= R2 and C1 >= C2, where P1 >= P2 indicates a higher resolution.
[0060] In addition, as shown, image sensor 202A and image sensor 202B each generate an intermediate frame image data object having the same intermediate frame image size, which represents a resolution falling between a first resolution defined by a first full frame image size and a second resolution defined by a second full frame image size. In this regard, the intermediate frame image size may represent a size that reduces the number of rows and / or columns from a larger full frame image size and increases the number of rows and / or columns from a smaller full frame image size. Returning to the above exemplary case, where R1>=R2 and C1>=C2, and for example, in the case where image sensor 202A has a higher resolution (e.g., (R1, C1) represents a higher image resolution than (R2, C2)), the intermediate frame image size may be represented by (Rm, Cm), where Rm represents the number of rows to be included in the intermediate frame image data object, and Cm represents the number of columns to be included in the intermediate frame image data object. In this regard, the intermediate frame image data object represents a third intermediate frame image resolution Pm, where Pm=Rm*Cm. In addition, the intermediate frame image size represents a third resolution that falls between the resolution represented by the first full frame image size and the resolution represented by the second full frame image size. In this regard, the intermediate frame image size represented by (Rm, Cm) can be defined so as to satisfy the relationship R1>=Rm>R2 and C1>Cm>=C2, and P1>Pm>P2. Non-limiting examples of intermediate frame image sizes include (R1, C1) and (R2, C2). In this regard, it should be understood that the above-determined formulas are merely examples. In some embodiments, the intermediate frame image size is limited only by the ability of each sensor to fill the number of rows and / or the number of columns in the image data object.
[0061] In other embodiments, it should be understood that the intermediate frame image size embodies any image size for standardized output on multiple image sensors. For example, in some embodiments, the intermediate frame image size embodies the maximum image size across all image sensors (e.g., where Rm is equal to the maximum number of rows of any full frame image size, and Cm is equal to the maximum number of rows of any full frame image size). In other embodiments, the intermediate frame image size embodies the minimum image size across all image sensors (e.g., where Rm is equal to the minimum number of rows of any full frame image size, and Cm is equal to the minimum number of rows of any full frame image size). In other embodiments, the minimum frame image size embodies any resolution between the maximum image size and the minimum image size. In some such embodiments, the selected intermediate frame image size may be determined by one or more components of the system (e.g., by a processor) or selected via user configuration, such that the determined or selected intermediate frame image size is used for specific hardware limitations and / or specific image processing tasks.
[0062] The image sensor may be configured to output intermediate frame image data objects based on the desired intermediate frame image size. In this regard, based on the various image sensors and processors, the intermediate frame image sizes are designed to take advantage of the resolution and bandwidth of the various components. In addition, the intermediate frame image sizes may also be designed based on the specific image processing tasks to be implemented via these components. In this regard, as the intermediate frame image size increases, more full-frame image data objects may be included for larger image sensors in exchange for a reduced transmission speed of the image data objects (e.g., a lower frame rate). Similarly, in at least some examples, as the intermediate frame image size decreases, fewer full-frame image data objects may be included in exchange for an increased transmission speed of the image data objects (e.g., a higher frame rate).
[0063] In an exemplary case, the intermediate frame image size may represent a particular region of interest within a larger full frame image data object (e.g., a region of interest within a full frame image data object captured by image sensor 202A), as well as any padding required to increase the number of rows and / or columns in the smaller full frame image data object (e.g., padding the full frame image data object captured by image sensor 202B). In one particular exemplary case, Rm=R2 and Cm=C1, such that the intermediate frame image size is represented by (R2, C1), where image sensor 202A captures a full frame image data object at (R1, C1), and image sensor 202B captures a full frame image data object at (R2, C2), where R1>R2 and C1>C2. In this regard, the intermediate frame image data object 204A may be generated by cropping a full frame image data object captured by the image sensor 202A and including relevant portions of the data (e.g., rows and / or columns of data within the region of interest) such that rows R1-Rm are cropped (e.g., (R1-Rm) / 2 are cropped from each of the top and bottom of the captured full frame image data object). Also in this regard, the intermediate frame image data object 204B may be generated by including the full frame image data object in the intermediate frame image data object 204B and filling the remaining data columns (e.g., filling the remaining Cm-C2 data columns, where the (Cm-C2) / 2 data columns are filled on each of the left and right sides of the intermediate frame image data object). It should be understood that such cropping may be used to identify and / or otherwise extract a specific portion of the full frame image data object representing the region of interest by cropping one or more rows and / or columns of data. Similarly, it should be understood that such padding may be included to fill any number of rows and / or columns of the intermediate frame image data object.
[0064] By reducing the number of rows and / or columns between the full-frame image data object and the intermediate-frame image data object 204A (e.g., by cropping a specific portion of the full-frame image data object to be included in the intermediate-frame image data object 204A), the intermediate-frame image data object 204A can be analyzed at a higher frame rate without sacrificing device usability. In addition, by increasing the number of rows and / or columns to standardize the intermediate-frame image size, image data objects for analysis are output in a standardized manner without requiring additional post-processing hardware to further configure and / or standardize the image data object format. Such exemplary advantages achieve improved image data object analysis throughput and efficacy in a cost-effective manner.
[0065] After configuring image sensors 202A and 202B based on the desired intermediate frame image size, image sensor 202A and image sensor 202B each generate a corresponding intermediate frame image data object based on the intermediate frame image size. For example, as shown, image sensor 202A can generate intermediate frame image data object 204A from the captured full frame image data object, and intermediate frame image data object 204A has an intermediate frame image size represented by (Rm, Cm). Similarly, image sensor 202B can generate intermediate frame image data object 204B from a second captured full frame image data object, and intermediate frame image data object 204B also has an intermediate frame image size represented by (Rm, Cm).
[0066] As shown, the image frame multiplexer 206 is configured to receive various input channels. Specifically, the image sensor 202A outputs the intermediate frame image data object 204A, and the image sensor 202B outputs the intermediate frame image data object 204B, wherein the image frame multiplexer 206 receives each intermediate frame image data object. Each intermediate frame image data object in the intermediate frame image data object is received via different input channels, so that any one of the input channels can be selected for output by the image frame multiplexer 206. The image frame multiplexer 206 is configured to output one of the input channels (e.g., the intermediate frame image data object 204A or the intermediate frame image data object 204B). The image frame multiplexer 206 receives a frame selection signal 208 for controlling the input channel for output. For example, as shown, the image frame multiplexer 206 outputs a selected image data object 210 based on the frame selection signal 208. The selected image data object 210 embodies one of the data objects received via the input channel (e.g., one of the intermediate frame image data objects 204A or 204B), and in particular, the data object received via the selected input channel indicated by the frame selection signal 208. In this regard, as shown, the selected image data object 210 embodies either the intermediate frame image data object 204A or the intermediate frame image data object 204B.
[0067] As shown, the processor 212 transmits the frame selection signal 208 to the image frame multiplexer 206. The processor 212 may generate the frame selection signal 208 for transmission to the image frame multiplexer 206. The processor 212 generates the frame selection signal 208 using a determined and / or predetermined frame selection algorithm. In some embodiments, the processor 212 is configured to utilize a random frame selection algorithm such that for each cycle, the generated frame selection signal 208 randomly selects between the input channels (e.g., as shown, randomly selects between the intermediate frame image data object 204A and the intermediate frame image data object 204B). Alternatively, in other embodiments, the processor 212 is configured to utilize a sequential frame selection algorithm such that for each cycle, the generated frame selection signal 208 selects between the input channels based on a defined order (e.g., as shown, each cycle switches between the intermediate frame image data object 204A and the intermediate frame image data object 204B). It will be appreciated that the random frame selection algorithm may be readily extended to select from any number of input channels (e.g., randomly selecting between X received intermediate frame image data objects in X different channels, where X is a number). Similarly, it will be appreciated that the sequential frame selection algorithm may be readily extended to select from any number of input channels (e.g., sequentially selecting from X received intermediate frame image data objects, where X is a number, before looping back to the beginning or reversing direction at the end of each cycle).
[0068] In other embodiments, the processor 212 processes one or more received data objects to generate the frame selection signal 208. For example, in some embodiments, the processor 212 processes the received selected image data object 210 received from the image frame multiplexer 206 to generate the frame selection signal 208. The processor 212 may process the selected image data object 210 to determine whether to continue to utilize the same value of the frame selection signal 208, or to change the value to select a different input channel. For example, in some embodiments, the processor 212 processes the selected image data object 210 to determine whether the data object includes sufficient image data to perform a particular image processing task. In some such embodiments, in the event that the processor 212 determines that the selected image data object 210 includes sufficient data, for example by successfully processing the image data and successfully completing at least one iteration or operation of the corresponding image processing task, the frame selection signal 208 may remain unchanged so that subsequent selected image data objects are received via the same input channel because such image data objects are sufficient for the task. Alternatively, in some such embodiments, in the event that the processor 212 determines that the selected image data object 210 does not include sufficient data, the frame selection signal 208 may be changed to select a different input channel. In some embodiments, based on such determination, the changed value is generated based on a random frame selection algorithm or a sequential frame selection algorithm, as described above. Alternatively, in some other embodiments, the changed value of the frame selection signal 208 is generated based on the selected image data object 210, for example to predict the input channel that will perform the best for the image processing task based on the analysis of the selected image data object 210. It should be understood that such predictive analysis may utilize a predictive algorithm, a machine learning model, or other image processing algorithm. For example, in at least one embodiment, the processor is configured to process the selected image data object 210 to determine that the selected image data object 210 does not include sufficient information for completing the corresponding image processing task, and to identify intermediate frame image data objects associated with different image sensors (e.g., image sensors of larger full frames or image sensors of smaller full frame image sizes) predicted to include sufficient information. Additionally or alternatively, in some embodiments, the processor is further configured to generate a frame selection signal 208 based on processing a selected image data object 210 even when the processor successfully completes the image processing task, and it should be understood that any such algorithm for generating a frame selection signal 208 based on processing a selected image data object 210 may be extended to generate an image selection signal for selecting from any number of input channels associated with any number of image sensors.
[0069] In addition or alternatively, to generate the frame selection signal 208, in some embodiments, the processor 212 is configured to perform the desired image processing task. For example, as shown, the processor 212 is configured to receive the selected image data object 210 from the image frame multiplexer 206 for processing associated with one or more image processing tasks. In an exemplary case, the processor 212 is configured to process the selected image data object 210 using one or more volume measurement algorithms (e.g., for determining the volume size of an object), visual mark detection algorithms (e.g., for barcode scanning and / or decoding), object detection algorithms, etc. In addition or alternatively, in some embodiments, the processor 212 is configured to perform one or more actions based on the results of processing the selected image data object 210, and / or transmit the results of the processing and / or the selected image data object 210 to one or more other components (e.g., a display or other processing hardware, not shown) when completed.
[0070] Example of an intermediate frame image data object with output and selection
[0071] Thus, having described the exemplary apparatus, components, and data flows of the exemplary embodiments of the present disclosure, the visualization of the generation and output of the intermediate frame image data object will now be described. It should be understood that the visualization includes the visualization that can be described above with respect to Figure 1 and / or Figure 2 A number of non-limiting example intermediate frame image data objects generated and / or output by the described apparatus and components.
[0072] Figure 3 An exemplary visualization of the functionality of an image frame multiplexer for selecting intermediate image data objects in a dual sensor environment according to at least some exemplary embodiments of the present disclosure is shown. In this regard, the exemplary visualization provides the above-mentioned Figure 2 The exemplary visualization of the intermediate frame image data object selection process. It should be understood that Figure 3 The specific image sizes and variable values shown are exemplary only and do not limit the scope and spirit of the embodiments described herein.
[0073] therefore, Figure 3A first full-frame image data object 302 and a second full-frame image data object 304 are shown. Specifically, the first full-frame image data object 302 has a full-frame image size represented by a full-frame column count 302A and a full-frame row count 302B. In this regard, as shown, the first full-frame image data object 302 has a 1920×1200 resolution, wherein the full-frame column count 302A is equal to 1920 and the full-frame row count 302B is equal to 1200. Similarly, the second full-frame image data object 304 has a full-frame image size represented by a full-frame column count 304A and a full-frame row count 304B. In this regard, as shown, the second full-frame image data object 304 has a 1280×800 resolution, wherein the full-frame column count 304A is equal to 1280 and the full-frame row count 304B is equal to 1200. It should be appreciated that the first image sensor (not shown) may be configured to capture a first full frame image data object 302 and the second image sensor (not shown) may be configured to capture a second full frame image data object 304 .
[0074] The image sensors configured to capture the first full-frame image data object 302 and the second full-frame image data object 304 may each be configured based on a particular intermediate frame image size. Specifically, each image sensor may be configured based on the same intermediate frame image size between the two image sensors, the intermediate frame image size represented by the intermediate frame column count 310A and the intermediate frame row count 310B. In this regard, the intermediate frame image size has a 1920×800 resolution, wherein the intermediate frame column count 310A is equal to 1920 (equal to the full-frame column count 302A of the first full-frame image data object 302), and the intermediate frame row count 310B is equal to 800 (equal to the full-frame row count 304B of the second full-frame image data object 304). Therefore, the intermediate frame image size represents a resolution between the 1920×1200 resolution of the first full-frame image data object 302 and the 1280×800 resolution of the second full-frame image data object 304.
[0075] As shown, the intermediate frame image size may be used to generate a first intermediate frame image data object 306 and a second intermediate frame image data object 308. Specifically, an image sensor (not shown) configured to capture the first full frame image data object 302 may be configured to generate and output the first intermediate frame image data object 306 based on the intermediate frame image size. In this regard, the first intermediate frame image data object 306 may be generated by cropping and / or padding the first full frame image data object 302 as necessary to match the full frame image size. As shown, the first full frame image data object 302 includes more rows than may be included in the intermediate frame image size, and thus a particular portion of the first full frame image data object 302 may be cropped to be included in the first intermediate frame image data object 306. Specifically, as shown, the first full-frame image data object 302 includes 400 more rows than the number of rows allowed by the intermediate frame row count 310B, and therefore 200 rows can be cropped from the top and bottom of the first full-frame image data object 302 (indicated by dashed or "dotted lines") to identify portions of data for inclusion in the corresponding first intermediate frame image data object 306. As shown, no columns are cropped or padded because the column counts 302A and 310A match. Therefore, after cropping the first full-frame image data object 302, the first intermediate frame image data object 306 is embodied by the region of interest embodied by the identified data portions.
[0076] Similarly, an image sensor (not shown) configured to capture the second full-frame image data object 304 may be configured to generate and output a second intermediate-frame image data object 308 based on the intermediate-frame image size. In this regard, the second intermediate-frame image data object 308 may be generated by cropping and / or padding the second full-frame image data object 304 as needed to match the full-frame image size. As shown, the full-frame image data object includes fewer columns than included in the intermediate-frame image size, so additional column padding must be included in the second intermediate-frame image data object 308. Specifically, the second full-frame image data object 304 includes 640 columns less than required by the intermediate-frame column count 310A. Therefore, 640 column padding may be identified to be included in the second intermediate-frame image data object 308, specifically, with 320 columns of padding data located to the left of the image data and 320 columns of padding data located to the right of the image data. As shown, no rows are cropped or padded because the row counts 304B and 310B match. Therefore, the second intermediate frame image data object 308 is embodied by the second full frame image data object 304 having sufficient column filling data.
[0077] As shown, each of the first intermediate frame image data object 306 and the second intermediate frame image data object 308 is output to the image frame multiplexer 206. Specifically, the first intermediate frame image data object 306 is output (e.g., by a first image sensor not shown) to the image frame multiplexer 206 in a first input channel assigned a channel value of 0. Similarly, the second intermediate frame image data object 308 is output (e.g., by a second image sensor not shown) to the image frame multiplexer 206 in a second input channel assigned a channel value of 1.
[0078] The image frame multiplexer 206 is configured to receive a frame selection signal 208, for example, from a processor (not shown). The frame selection signal 208 reflects the value of one of the input channels, so that the frame selection signal 208 represents the input channel selected for output by the image frame multiplexer 206. As shown, the frame selection signal 208 represents a value of 0, indicating that the input channel represented by the value 0 is selected for output by the image frame multiplexer 206. Therefore, the image frame multiplexer 206 outputs the data object received via the first input channel represented by channel 0. Therefore, the frame selection signal 208 can be changed to select whether to output the first intermediate frame image data object 306 or the second intermediate frame image data object 308 without activating and / or deactivating any corresponding image sensor.
[0079] It should be understood that in some embodiments, a selected intermediate frame image data object may be analyzed to determine the channel and / or image sensor from which the data originates. For example, in an embodiment in which a selected intermediate frame image data object is output to a processor, the processor may identify the image sensor from which the selected intermediate frame image data object originates based on the amount of padding data. For example, as shown, the first intermediate frame image data object 306 may be processed to determine that the first intermediate frame image data object 306 originates from the first image sensor based on the absence of padding data. Similarly, if the second intermediate frame image data object 308 is selected for output, the processor may identify padding data within the second intermediate frame image data object 308 to determine that the data object originates from the second image sensor. In some such embodiments, the processor may store and / or otherwise access information associated with each image sensor (e.g., the full frame image size associated with each image sensor) to make such determinations. It should be understood that such processing may be used to identify an intermediate frame image data object originating from any one of a plurality of image sensors, such as as described below with respect to Figure 4 described.
[0080] Figure 4 A plurality of full-frame image data objects and corresponding intermediate-frame image data objects are shown, each corresponding intermediate-frame image data object being associated with a particular intermediate-frame image size that is the same between the intermediate-frame image data objects. In this regard, Figure 4 It can be associated with a multi-sensor environment such that each of the illustrated full-frame image data objects is captured by a different image sensor that is configured to generate a corresponding intermediate-frame image data object for output to an image frame multiplexer. It should be understood that the number of full-frame image data objects and corresponding intermediate-frame image data objects and the illustrated image sizes are merely exemplary.
[0081] Figure 4 A plurality of full-frame image data objects 402A to 402D are depicted, each having a different resolution. The full-frame image data objects 402A to 402D are each associated with a corresponding intermediate-frame image data object of a plurality of intermediate-frame image data objects 404A to 404D. Each of the intermediate-frame image data objects 404A to 404D is associated with the same intermediate-frame image size. Specifically, the intermediate-frame image size is 1920×1050 resolution, with an intermediate-frame column count of 1920 and an intermediate-frame row count of 1050. In this regard, each of the intermediate-frame image data objects 404A to 404D may be generated by clipping data from the corresponding full-frame image data object and / or padding data to the corresponding full-frame image data object.
[0082] For example, Figure 4 A first full frame image data object 402A and a corresponding first intermediate frame image data object 404A are shown. The first full frame image data object 402A has a first full frame image size, represented by a first full frame column count of 1920 and a first full frame row count of 1200. Thus, the first full frame image data object is associated with a full frame column count that matches the intermediate frame column count, such that no padding or cropping is required in that direction. The first full frame image data object 402A is associated with a full frame row count that exceeds the intermediate frame row count (1200 compared to 1050). Thus, the first intermediate frame image data object 404A includes a cropped image data portion 406A, wherein the cropped image data portion 406A includes a data portion of the first full frame image data object 402A after cropping a plurality of data rows (indicated by the area cropped by the dashed or "dotted" lines as shown). Specifically, the cropped image data portion 406A includes data of the first full frame image data object 402A having 75 lines cropped from each of the top and bottom of the first full frame image data object 402A (e.g., such that the total cropped lines equal the difference between the two line counts of 150).
[0083] In addition, for example, Figure 4A second full frame image data object 402B and a corresponding second intermediate frame image data object 404B are shown. The second full frame image data object 402B has a second full frame image size, represented by a second full frame column count of 1680 and a second full frame row count of 1050. Thus, the second full frame image data object 402B is associated with a full frame row count that matches the intermediate frame row count, such that no row padding or cropping is required in that direction. The second full frame image data object 402B is associated with a full frame column count that is less than the intermediate frame column count (1680 compared to 1920). Thus, the second intermediate frame image data object 404B includes the second full frame image data object 402B as well as the necessary image data padding 406B for the additional columns. Specifically, the second intermediate frame image data object 404B includes the second full frame image data object 402B and an image data fill 406B of 120 columns of fill data on each of the left and right sides of the second full frame image data object 402B (e.g., such that the total image data fill 406B is equal to the difference 240 between the two column counts).
[0084] In addition, for example, Figure 4 A third full frame image data object 402C and a corresponding third intermediate frame image data object 404C are shown. The third full frame image data object 402C has a third full frame image size, represented by a third full frame column count of 1440 and a third full frame row count of 900. The third full frame image data object 402C is associated with a full frame column count that is less than the intermediate frame column count (1440 compared to 1920). Additionally, the third full frame image data object 402C is associated with a full frame row count that is less than the intermediate frame row count (900 compared to 1050). Thus, the third intermediate frame image data object 404C includes the third full frame image data object 402C and the necessary image data padding 406C for the additional columns and rows. Specifically, the third intermediate frame image data object 404C includes the third full-frame image data object 402C and 240 columns of fill data on each of the left and right sides of the third full-frame image data object 402C (e.g., so that the total fill columns of the image data fill 406C are equal to the difference 480 between the two column counts) and 150 rows of fill data on each of the top and bottom of the third full-frame image data object 402C (e.g., so that the total fill rows of the image data fill 406C are equal to the difference 300 between the two row counts).
[0085] In addition, for example, Figure 4A fourth full frame image data object 402D and a corresponding fourth intermediate frame image data object 404D are shown. The fourth full frame image data object 402D has a fourth full frame image size, represented by a fourth full frame column count of 1280 and a fourth full frame row count of 800. The fourth full frame image data object 402D is associated with a full frame column count that is less than the intermediate frame column count (1280 compared to 1920). Additionally, the fourth full frame image data object 402D is associated with a full frame row count that is less than the intermediate frame row count (800 compared to 1200). Thus, the fourth intermediate frame image data object 404D includes the fourth full frame image data object 402D and the necessary image data padding 406D for the additional columns and rows. Specifically, the fourth intermediate frame image data object 404D includes the fourth full-frame image data object 402D and 320 columns of fill data on each of the left and right sides of the fourth full-frame image data object 402D (e.g., so that the total fill columns of the image data fill 406D are equal to the difference 640 between the two column counts) and 200 rows of fill data on each of the top and bottom of the fourth full-frame image data object 402D (e.g., so that the total fill rows of the image data fill 406D are equal to the difference 400 between the two row counts).
[0086] Each of the intermediate frame image data objects 404A to 404D may be output by a corresponding image sensor to an image frame multiplexer for selection. Similarly, it should be understood that each of the intermediate frame image data objects 404A to 404D may be processed to identify the image sensor from which the intermediate frame image data object originated. For example, by determining the number of padded rows and padded columns in a particular intermediate frame image data object, the processor may determine the resolution of the original image from a set of known resolutions, each resolution associated with a particular image sensor. In this regard, the intermediate frame image data objects may be used to generate a new frame selection signal, or to request data directly from the corresponding image sensor (e.g., requesting a full frame image data object from the sensor when the processor determines that the intermediate frame image data object is missing important data due to, for example, cropping).
[0087] It should be understood that in other embodiments, the intermediate frame image data object can be associated with a different intermediate frame image size. For example, in some embodiments, the intermediate frame image size can represent the maximum number of rows and columns of all full-frame image sizes (e.g., 1920×1200 as shown), so that each intermediate frame image data object can be generated by filling each smaller full-frame image data object in each direction. In addition, in other embodiments, the intermediate frame image size can represent the minimum number of rows and columns of the full-frame image size (e.g., 1280×800 as shown), so that each intermediate frame image data object can be generated by cropping a specific area of each smaller full-frame image data object.
[0088] Exemplary Process for Intermediate Frame Image Processing
[0089] Various exemplary devices, components, data flows, and various exemplary full-frame image data objects with corresponding intermediate frame image data objects have been described according to exemplary embodiments of the present disclosure, and exemplary processes for image processing using intermediate frame image data objects (referred to as "intermediate frame image processing") are shown. In this regard, it should be understood that the processes described herein can be implemented as computer-implemented processes via the devices and / or specific components described herein. In addition or alternatively, it should be understood that the processes may include one or more optional operations (shown in dashed or "dotted lines"). In some embodiments, no optional operations, some optional operations, or all optional operations are performed within the process.
[0090] Figure 5 An exemplary flow chart including exemplary operations for intermediate frame image processing according to at least some exemplary embodiments of the present disclosure is shown. The exemplary operations embody a process (or "method") 500. In some embodiments, it should be understood that the exemplary operations embody the process of processing an intermediate frame image by an apparatus (e.g., as described above with respect to Figure 1 The intermediate frame imaging device 100) and / or one or more hardware components (for example, the above relative to Figure 2 A computer-implemented process performed by those hardware components described above.
[0091] At optional block 502, process 500 includes configuring, by the processor, at least one image sensor of the plurality of image sensors based on the intermediate frame image size. In this regard, the processor may configure the at least one image sensor to generate and output an intermediate frame image data object based on the intermediate frame image size. In some embodiments, at optional block 502, all of the plurality of image sensors are configured. In other embodiments, only a subset of the plurality of image sensors are configured, e.g., where one or more image sensors capture a full frame image data object associated with a full frame image size that matches the intermediate frame image size. In some embodiments, using the method described below with respect to Figure 7 In other embodiments, at least one (or all) of the plurality of image sensors are configured using computing hardware other than a processor (eg, external computing hardware).
[0092] At block 504, process 500 includes outputting a plurality of intermediate frame image data objects from a plurality of image sensors to an image frame multiplexer. Each of the plurality of intermediate frame image data objects is associated with an intermediate frame image size, such that each of the plurality of image sensors generates an intermediate frame image data object of the intermediate frame image size, regardless of the full frame image size of the full frame image data object captured by each image sensor. In some embodiments, each image sensor outputs a corresponding intermediate frame image data object to a different input channel of the image frame multiplexer. It should be understood that in at least some embodiments, the plurality of image sensors are embodied by the same plurality of image sensors configured at optional block 502. In some embodiments, block 504 includes setting each of the plurality of image sensors to a particular mode so that the plurality of image sensors output intermediate frame image data objects. In some embodiments, the processor is configured to directly connect the plurality of image sensors to configure a sensor region of interest. Additionally or alternatively, in some embodiments, the processor and / or each image sensor is configured to capture a full frame image data object, and then crop and / or fill the image data based on one or more software implementations.
[0093] At box 506, process 500 includes selecting a selected intermediate frame image data object from a plurality of intermediate frame image data objects by an image frame multiplexer. In some such embodiments, the image frame multiplexer receives a frame selection signal for selecting a specific input channel, each input channel being associated with one of the received intermediate frame image data objects. In this regard, the image frame multiplexer is configured to select an input channel based on the frame selection signal, identify the intermediate frame image data object associated with the selected input channel, and output the identified intermediate frame image data object. For example, the image frame multiplexer may receive a frame selection signal from a processor for selecting a selected intermediate frame image data object from a plurality of intermediate frame image data objects based on the frame selection signal by switching to enable only a single input channel to be connected to an output channel. By selecting the selected intermediate frame image data object, the image frame multiplexer is configured to output the selected intermediate frame image data object via an output channel. In other embodiments, the image frame multiplexer receives a frame selection signal from a controller hardware, which is configured to generate a frame selection signal and / or otherwise transmit the frame selection signal to the image frame multiplexer. In some embodiments, the controller hardware associated with the image frame multiplexer can communicate with the processor to receive an update signal and / or determine the value of the frame selection signal and / or determine when to update the frame selection signal. In other embodiments, the controller hardware associated with the image frame multiplexer is independent of the processor.
[0094] At block 508, process 500 includes receiving, by the processor, the selected intermediate frame image data object from the image frame multiplexer. In this regard, the processor may be connected to receive the data object output via an output channel of the image frame multiplexer. It should be understood that in this regard, the image frame multiplexer may transmit the selected intermediate frame image data object to the processor via any of a variety of known communication methods, including via a hardwired connection, an alternative wired connection, and / or a wireless connection.
[0095] At block 510, process 500 includes processing the selected intermediate frame image data object by the processor. In some such embodiments, the processor is configured to process the selected intermediate frame image data object using any of a plurality of image processing algorithms. The processor may process the selected intermediate frame image data object for any of a plurality of purposes. For example, in some embodiments, the processor may process the selected intermediate frame image data object for a particular image processing task. Additionally or alternatively, in some embodiments, the processor may process the selected intermediate frame image data object to determine whether there is significant data missing (e.g., due to cropping) in the selected intermediate frame image data object in order to determine whether to request a full frame image data object. Additionally or alternatively, in some embodiments, the processor may process the selected intermediate frame image data object for updating a frame selection signal. It should be understood that any number of computer-implemented algorithms may be utilized to perform each processing task.
[0096] At optional block 512, process 500 includes updating a frame selection signal that controls the image frame multiplexer by a processor. The processor may generate the updated frame selection signal based on one or more frame selection algorithms. For example, in some embodiments, the processor is configured to update the frame selection signal using a random frame selection algorithm or a sequential frame selection algorithm. In addition or alternatively, in some embodiments, the processor is configured to update the frame selection signal based on the result of processing the selected intermediate frame image data object. For example, in some embodiments, the processor processes the selected intermediate frame image data object to determine whether the data therein is sufficient for the image processing task, and / or whether another image sensor associated with a different input channel is predicted to provide a more useful image. In this regard, based on such analysis, the processor may update the frame selection signal to reflect the input channel predicted to receive more useful data. In other embodiments, the processor is configured to update the controller hardware associated with the image frame multiplexer so that the controller hardware updates the frame selection signal.
[0097] Figure 6 An exemplary flowchart according to at least some exemplary embodiments of the present disclosure is shown, the exemplary flowchart including exemplary additional operations for intermediate frame image processing, specifically including additional operations for requesting and processing selected full frame image data objects associated with selected intermediate frame image data objects. The exemplary operations embody a process (or "method") 600. In some embodiments, it should be understood that the exemplary operations embody the process of processing by an apparatus (e.g., as described above with respect to Figure 1 The intermediate frame imaging device 100) and / or one or more hardware components (for example, the above relative to Figure 2 A computer-implemented process performed by those hardware components described above.
[0098] In at least some embodiments, process 600 occurs after one or more blocks of process 500. In this regard, process 600 may begin after block 510 or optional block 512, as described above with respect to FIG. Figure 5 In at least some embodiments, process 600 occurs as a result of processing the intermediate frame image data object at block 510 to make one or more determinations. For example, in some embodiments, process 600 occurs in response to processing a selected intermediate frame image data object by a processor to determine that useful data is missing from the selected intermediate frame image data object (e.g., data that has been clipped as being necessary and / or helpful in completing an image processing task effectively or efficiently).
[0099] At block 602, process 600 includes identifying, by a processor, a selected image sensor from a plurality of image sensors. The processor identifies the selected image sensor based at least on a selected intermediate frame image data object. For example, in some embodiments, the processor identifies the selected image sensor based on fill data and / or image data within the selected intermediate frame image data object, wherein such data is used to link the selected intermediate frame image data object to a resolution associated with the selected image sensor. In some embodiments, the processor is configured to store and / or otherwise access resolution data, configuration data, and / or other information associated with each of the plurality of image sensors for identifying the selected image sensor based on the selected intermediate frame image data object. In some embodiments, the processor is configured to communicate directly with each of the plurality of image sensors to receive resolution data, configuration data, etc. for comparison with the selected intermediate frame image data object. In some other embodiments, the processor is configured to communicate with a data storage device (e.g., a memory device) to retrieve such data.
[0100] At block 604, process 600 includes transmitting, by the processor, a full frame request signal to the selected image sensor. The full frame request signal includes specially configured information indicating a request to receive a full frame image data object associated with the selected intermediate frame image data object or a subsequently captured full frame image data object from the selected image sensor. In some embodiments, the processor transmits the full frame request signal directly (e.g., via one or more wired connections between the selected image sensor and the processor). In other embodiments, the processor transmits the full frame request signal indirectly via one or more intermediate components.
[0101] In some embodiments, the full frame request signal is configured to cause the selected image sensor to transmit the full frame request signal from a buffer, a temporary storage device, or a permanent storage device. In other embodiments, the full frame request signal is configured to cause the selected image sensor to transmit a subsequently captured full frame image data object. In addition or alternatively, in some embodiments, the full frame request signal causes the selected image sensor to change one or more modes that facilitate the transmission of the full frame image data object. For example, in some embodiments, the full frame request signal causes the selected image sensor to change the output mode to a mode that outputs the captured full frame image data object instead of the corresponding intermediate frame image data object. Such a mode may last until a subsequent signal is received, for a limited amount of time, or for multiple captures.
[0102] In some embodiments, the full frame request signal causes the selected image sensor to be reconfigured to output a full frame image data object instead of an intermediate frame image data object. For example, in this regard, the full frame request signal may reformat the selected image sensor based on a full frame image size. In some embodiments, the selected image sensor is caused to capture a full frame image data object after the selected image sensor is reconfigured.
[0103] At block 606, process 600 includes receiving, by the processor, a selected full-frame data object from the selected image sensor. It should be understood that the full-frame image data object includes more data of the selected intermediate frame image data object. In this regard, in at least one exemplary case, when the selected intermediate frame image data object is generated, the full-frame image data object includes previously cropped image data. In other embodiments, the full-frame image data object includes new full-frame image data captured by the selected image sensor. Therefore, the selected full-frame image data object may include data that can be used for image processing tasks performed by the processor, where useful data was previously cropped. In this regard, in the case where the intermediate frame image data object is insufficient to successfully (or effectively) complete the image processing task, the corresponding selected full-frame image may be received and processed for such purpose. In some cases, if the number of full-frame request signals being issued exceeds a certain threshold, it may be necessary to increase the size of the intermediate frame image to optimally complete the image processing task. In some such embodiments, the processor may automatically perform such intermediate frame image size reconfiguration, or an operator may perform such intermediate frame image size reconfiguration.
[0104] In some embodiments, the processor receives the selected full-frame image data object via the image frame multiplexer. For example, in some embodiments, upon transmission of the full-frame request signal (e.g., at block 604), the processor configures the image frame multiplexer to continue outputting from the input channel associated with the selected image sensor such that the image frame multiplexer continues to output data from the input channel associated with the selected image sensor for the duration of the frame capture. In this regard, the image frame multiplexer continues to output the full-frame image data object from the selected image sensor until the selected full-frame image data object is received and / or processed by the processor.
[0105] At block 608, process 600 includes processing, by the processor, the selected full-frame image data object. In doing so, the processor is able to perform image processing tasks based on all captured data such that previously cropped data is restored and available for analysis. In this regard, processing the selected full-frame image data object may enable successful completion of the image processing task, or improve the accuracy of such results. By minimizing the number of times the selected full-frame image data object is requested (and therefore, where practical, relies on intermediate frame image data objects), embodiments of the present disclosure may maintain improved average throughput while enabling access to full-frame image data objects in situations that would otherwise result in errors or reduced efficiency and / or efficacy.
[0106] Figure 7 An exemplary flowchart according to at least some exemplary embodiments of the present disclosure is shown, the exemplary flowchart including exemplary additional operations for intermediate frame image processing, specifically including exemplary additional operations for configuring one or more image sensors based on the intermediate frame image size. The exemplary operations embody the process (or "method") 700. In some embodiments, it should be understood that the exemplary operations embody the process (e.g., the process described above with respect to the intermediate frame image size) of the device ... Figure 1 The intermediate frame imaging device 100) and / or one or more hardware components (for example, the above relative to Figure 2 A computer-implemented process performed by those hardware components described above.
[0107] In at least some embodiments, process 700 occurs after or in place of one or more blocks of process 500. In this regard, process 700 may replace the process described above with respect to Figure 5 Upon completion of process 700 or a subset of the operations therein, the flow may proceed to one or more other operations of another process. For example, as shown, the flow may return to block 504 of process 500, as described above with respect to Figure 5 described.
[0108] At block 702, process 700 includes identifying, by a processor, a plurality of full-frame image sizes associated with a plurality of image sensors. In some embodiments, the processor may communicate directly with each of the plurality of image sensors. In some such embodiments, the processor may request resolution data representing a full-frame image size from each of the plurality of image sensors. Additionally or alternatively, in some embodiments, the processor is configured to identify a full-frame image size for each of the plurality of image sensors by receiving a full-frame image data object from the corresponding image sensor, directly or indirectly through one or more other components (e.g., an image frame multiplexer), and determining the full-frame image size from each full-frame image data object. In other embodiments, the processor accesses the plurality of full-frame image sizes from a temporary or permanent data storage device (e.g., embodied by one or more memory devices).
[0109] At block 704, process 700 includes determining, by the processor, a maximum image size based on the plurality of full-frame image sizes. Similarly, at block 706, process 700 includes determining, by the processor, a minimum image size based on the plurality of full-frame image sizes. In some embodiments, the processor is configured to determine the maximum image size and / or the minimum image size based on a comparison between various full-frame image sizes in the plurality of full-frame image sizes. It should be understood that any known algorithm for determining the maximum and / or minimum image size in the plurality of image sizes may be used to determine the minimum image size and / or the maximum image size from the plurality of full-frame image sizes.
[0110] At block 708, process 700 includes generating, by the processor, an intermediate frame image size based on the maximum image size and the minimum image size. In some embodiments, the intermediate frame image size is generated to fall between the maximum image size and the minimum image size, and based on the desired throughput and / or image processing tasks. In this regard, for example, the intermediate frame row count may be generated based on the full frame row count of the maximum image size, and the intermediate frame column count may be generated based on the full frame column count of the minimum image size (or vice versa), wherein the intermediate frame row count and the intermediate frame column count form the intermediate frame image size. In other embodiments, the intermediate frame image size is selected based on the row count and / or column count of one or more intermediate image sizes of a plurality of full frame image sizes (e.g., full frame image sizes between the maximum image size and the minimum image size).
[0111] In other embodiments, the maximum full-frame image size of any image sensor is used as the intermediate frame image size. In some such embodiments, each other image sensor of the plurality of image sensors may be configured to pad the captured image data object to extend the captured image data object to match the intermediate frame image size. In other embodiments, the minimum full-frame image size of any image sensor is used as the intermediate frame image size. In some such embodiments, each other image sensor of the plurality of image sensors may be configured to crop the relevant portion of the captured image data to match the intermediate frame image size in one or more directions.
[0112] At box 710, process 700 includes configuring, by a processor, at least one of the plurality of image sensors to output an intermediate frame image data object based on the intermediate frame image size. In some embodiments, the processor transmits the intermediate frame image size to each of the at least one of the plurality of image sensors and / or transmits a configuration signal indicating that the received data should be used to set one or more modes and / or stored for use in generating future intermediate frame image data objects. For example, in some embodiments, each image sensor is configured to activate a particular output mode associated with outputting the intermediate frame image data object in response to receiving the intermediate frame image size. Additionally or alternatively, each image sensor that receives the intermediate frame image size stores the intermediate frame image size for use in subsequent intermediate frame image data object generation.
[0113] Figure 8 An exemplary flowchart according to at least some exemplary embodiments of the present disclosure is shown, the exemplary flowchart including exemplary additional operations for intermediate frame image processing, specifically including exemplary additional operations for generating and outputting intermediate frame image data objects. The exemplary operations embody the process (or "method") 800. In some embodiments, it should be understood that the exemplary operations embody the process (e.g., the process described above with respect to the intermediate frame image data object) of the device ... Figure 1 For example, in some embodiments, the image sensor and / or corresponding supporting circuitry of the intermediate frame imaging device performs a computer-implemented process relative to Figure 8 It should be understood that each image sensor in the plurality of image sensors can be configured to perform process 900 independently of the other image sensors. It should be understood that process 900 can be repeated independently by any number of image sensors.
[0114] At block 802, process 800 includes configuring an image sensor based on the received intermediate frame image size. In this regard, in some embodiments, the image sensor is configured to capture an intermediate frame image data object based on the intermediate frame image size. In some embodiments, the image sensor is configured to perform one or more image data manipulation operations (e.g., cropping and / or padding) when capturing the intermediate frame image data object.
[0115] At block 804, process 800 includes capturing an intermediate frame image data object based on the received intermediate frame image size and the full frame image size. The full frame image size may be associated with an image sensor, for example, where the image sensor is configured to capture a full frame image data object of the full frame image size. In this regard, to capture the intermediate frame image data object, the image sensor may crop and / or pad the data captured by the image sensor. For example, the intermediate frame image data object may be captured based on the difference between the full frame image size and the configured intermediate frame image size such that sufficient data is cropped from the captured data and / or sufficient data is padded to the captured data. In some such embodiments, as described below with respect to Fig.10 The method is used to perform cropping and / or padding of the captured data.
[0116] At block 806, process 800 includes transmitting the intermediate frame image data object to the image frame multiplexer. In some such embodiments, for example, the image sensor is configured to have an output channel that is connected (e.g., wired) to an input channel of the image frame multiplexer. In this regard, the image sensor can transmit the intermediate frame image data object via such a connection so that the intermediate frame image multiplexer receives the intermediate frame image data object via a defined input channel associated with the sensor. Multiple image sensors can each utilize various input channels to transmit the intermediate frame image data objects generated by them to the image frame multiplexer, which can be selected from these input channels by the image frame multiplexer as described herein.
[0117] Fig. 9 An exemplary flowchart according to at least some exemplary embodiments of the present disclosure is shown, which includes exemplary additional operations for intermediate frame image processing, specifically including a specific implementation for generating and outputting intermediate frame image data objects. The exemplary operations embody a process (or "method") 900. In some embodiments, it should be understood that the exemplary operations embody the process of processing an intermediate frame image by an apparatus (e.g., as described above with respect to Figure 1 For example, in some embodiments, the image sensor and / or corresponding supporting circuitry of the intermediate frame imaging device performs a computer-implemented process relative to Fig. 9It should be understood that each image sensor in the plurality of image sensors can be configured to perform process 900 independently of the other image sensors. It should be understood that process 900 can be repeated independently by any number of image sensors.
[0118] At block 902, process 900 includes capturing a full frame image data object associated with a full frame image size. It should be appreciated that the image sensor may be configured to capture the full frame image data object using any of a variety of known methods. Further, in this regard, it should be appreciated that the image sensor may capture the full frame image data object using different methods based on the specific implementation of the image sensor (e.g., CCD, CMOS, etc.). In some embodiments with multiple image sensors, any number of image sensor types may be used (e.g., all of the same image sensor type, all of different image sensor types, or some image sensors of a particular image sensor type and some other image sensor types).
[0119] At block 904, process 900 includes identifying an intermediate frame image size. In this regard, the image sensor may have been previously configured to have an intermediate frame image size. For example, the image sensor may have been previously configured to have an intermediate frame image size as described above with respect to Figure 7 In this regard, the image sensor and / or corresponding supporting circuitry may identify the intermediate frame image size from a data storage device capable of communicating with such components.
[0120] At box 906, process 900 includes identifying a cropped image data portion of a full-frame image data object based on the full-frame image size and the intermediate-frame image size. The cropped image data portion represents a specific area of interest within a full-frame image size that matches or is smaller than the intermediate-frame image size. In some embodiments, the cropped image data portion of a full-frame image data object is identified by cropping at least one data row from the full-frame image data object and / or cropping at least one data column from the full-frame image data object. It should be understood that if the full-frame image size of the full-frame image data object is smaller than the intermediate-frame image size in both the number of columns and the number of rows, the cropped image data portion may represent the entire full-frame image data object. In this document with respect to Fig. 9 The described operations provide an exemplary process for identifying a cropped image data portion of a full frame image data object based on the full frame image size and the intermediate frame image size.
[0121] At block 908, process 900 includes identifying image data padding based on the full frame image size and the intermediate frame image size. The image data padding includes the number of rows and / or columns to be filled with padding data. In this regard, the image padding data may be used to fit the difference between the full frame image size and the intermediate frame image size. Fig. 9The described operations provide an exemplary process for identifying image data padding based on full frame image size and intermediate frame image size.
[0122] At box 910, process 900 includes generating an intermediate frame image data object based on the cropped image data portion and the image data padding of the full frame image data object. In some such embodiments, the intermediate frame image data object is generated to include the cropped image data portion centered within the intermediate frame image data object, wherein the image data padding is evenly distributed around the cropped image data portion. In this regard, any image data padding can be evenly divided between the respective sides of the intermediate frame image data object (e.g., half of the row image data padding is at the top of the intermediate frame image data object, half of the row image data padding is below the intermediate frame image data object, half of the column image data padding is on the left side of the intermediate frame image data object, and half of the column image data padding is on the right side of the intermediate frame image data object). In some embodiments, for example, where the cropped image data portion includes a number of rows and columns that match the size of the intermediate frame image, the intermediate frame image data object is embodied by the cropped image data portion.
[0123] At block 912, process 900 includes transmitting the intermediate frame image data object to the image frame multiplexer. In some such embodiments, for example, the image sensor is configured to have an output channel that is connected (e.g., wired) to an input channel of the image frame multiplexer. In this regard, the image sensor may transmit the intermediate frame image data object via such a connection so that the intermediate frame image multiplexer receives the intermediate frame image data object via a defined input channel associated with the sensor. Multiple image sensors may each utilize various input channels to transmit their generated intermediate frame image data objects to the image frame multiplexer, which may be selected from these input channels by the image frame multiplexer as described herein.
[0124] Fig.10 An exemplary flowchart according to at least some exemplary embodiments of the present disclosure is shown, the exemplary flowchart including exemplary additional operations for intermediate frame image processing, specifically including exemplary additional operations for identifying cropped image data portions and / or image data fills for specific full-frame image data objects. The exemplary operations embody the process (or "method") 1000. In some embodiments, it should be understood that the exemplary operations embody the process (e.g., the above with respect to Figure 1 For example, in some embodiments, the image sensor and / or corresponding supporting circuitry of the intermediate frame imaging device performs a computer-implemented process relative to Fig.10It should be understood that each image sensor in the plurality of image sensors can be configured to perform process 1000 independently of the other image sensors. It should also be understood that process 1000 can be repeated independently by any number of image sensors.
[0125] In at least some embodiments, process 1000 occurs after or in place of one or more blocks of process 1000. In this regard, process 1000 may replace the above with respect to Fig. 9 Upon completion of process 1000 or a subset of the operations therein, the flow may proceed to one or more other operations of another process. For example, as shown, the flow may return to block 910 of process 1000, as described above with respect to Fig. 9 described.
[0126] At block 1002, process 1000 includes identifying a cropped image data portion of a full-frame image data object based on a difference between an intermediate frame row count and a full-frame row count and / or a difference between an intermediate frame column count and a full-frame column count. In this regard, the cropped image data portion may be identified from the full-frame image data object by cropping out the difference in rows and / or columns between the full-frame image size and the intermediate frame image size. It should be appreciated that in some examples, only the number of data rows is cropped from the full-frame image data object, and in other cases, only the number of data columns is cropped from the full-frame image data object, and in still other cases, both the number of data rows and the number of data columns are cropped from the full-frame image data object.
[0127] When cropping a full-frame image data object, the difference between the row count and / or column count may be evenly cropped from the sides of the full-frame image data object. In this regard, the difference between the intermediate frame row count and the full-frame row count may be evenly cropped from the top of the full-frame image data object and the bottom of the full-frame image data object, such that half of the difference is cropped from each side. In a specific non-limiting example where the row difference value is 200, 100 data rows may be cropped from the top of the full-frame image data object, and 100 data rows may be cropped from the bottom of the full-frame image data object. Similarly, the difference between the intermediate frame column count and the full-frame column count may be evenly cropped from the left side of the full-frame image data object and the right side of the full-frame image data object, such that half of the difference is cropped from each side. In a specific non-limiting example where the column difference value is 240, 120 data columns may be cropped from the left side of the full-frame image data object, and 120 data columns may be cropped from the right side of the full-frame image data object. After the data rows are cropped, the remaining data rows represent the cropped image data portion of the full-frame image data object.
[0128] In other implementations, other cropping methods may be used based on specific areas of interest. In some embodiments, a certain percentage of data may be cropped from each side based on the image processing task to be completed. For example, in a first non-limiting example, 100% of the data rows may be cropped from the top of a full-frame image data object, where the image processing task may require the bottom of the captured full-frame image data object. Alternatively, in another non-limiting example, 75% may be cropped from the left side of the full-frame image data object, and 25% may be cropped from the right side, where the image processing task may include useful data in the middle to middle-right portion of the full-frame image data object.
[0129] In the case where the full frame row count is not greater than the intermediate frame row count and the full frame column count is not greater than the intermediate frame column count, it should be understood that no cropping of the captured data is required. In this regard, the full frame image size is equal to or less than the intermediate frame image size in both directions (e.g., in rows and columns). Therefore, the cropped image data portion includes the captured data without cropping and therefore represents the entirety of the full frame image data object to be included in the generated intermediate frame image data object.
[0130] In some embodiments, process 1000 includes one or more determination steps as to whether the captured data should be cropped. For example, in at least some embodiments, process 1000 includes determining whether the full frame row count is greater than the intermediate frame row count, and / or whether the full frame column count is greater than the intermediate frame column count. In this regard, the full frame data object captured by the image sensor is associated with a full frame image size representing a full frame row count (indicating the number of data rows in the full frame image data object) and a full frame column count (indicating the number of data columns in the full frame image data object). Similarly, the intermediate frame image size represents an intermediate frame row count and an intermediate frame column count, the intermediate frame row count indicating the number of data rows to be included in the corresponding intermediate frame image data object, and the intermediate frame column count indicating the number of data columns to be included in the corresponding intermediate frame image data object. Therefore, the full frame row count can be compared with the intermediate frame row count to determine whether the full frame row count is greater than the intermediate frame row count, or equal to or less than the intermediate frame row count. Similarly, the full frame column count can be compared with the intermediate frame column count to determine whether the full frame column count is greater than the intermediate frame column count. In some embodiments, an image sensor and / or supporting hardware for performing such comparisons are provided. Process 1000 may continue to identify the cropped image data portion based on the comparison. For example, in the case where the full frame row count is greater than the intermediate frame row count, the full frame column count is greater than the intermediate frame column count, or both, process 1000 includes identifying the cropped image data portion of the full frame image data object based on the difference between the intermediate frame row count and the full frame row count and / or the difference between the intermediate frame column count and the full frame column count. In the case where the full frame row count is not greater than the intermediate frame row count and the full frame column count is not greater than the intermediate frame column count, process 1000 includes identifying the cropped image data portion that includes the entirety of the full frame image data object.
[0131] At block 1004, process 1000 includes identifying image data padding based on a difference between the intermediate frame row count and the full frame row count and / or a difference between the intermediate frame column count and the full frame column count. In this regard, a difference between the intermediate frame row count and the full frame row count (e.g., Rp=Rm-Rf, where Rp is the image data padding row count, Rm is the intermediate frame row count, and Rf is the full frame row count) may be identified for use as padding in a first direction (e.g., vertical padding on the top and / or bottom of an intermediate frame image data object). Similarly, in this regard, a difference between the intermediate frame column count and the full frame column count (e.g., Cp=Cm-Cf, where Cp is the image data padding column count, Cm is the intermediate frame column count, and Cf is the full frame column count) may be identified for use as padding in a second direction (e.g., horizontal padding on the left and / or right side of an intermediate frame image data object). In this regard, the image data padding may embody a combination of an image data padding column count and an image data padding row count (eg, (Cp, Rp)).
[0132] It should be understood that padding can be identified based on the specific distribution of each edge of the intermediate frame image data object. For example, in some embodiments, the image data padding includes various values representing the padding of each edge of the corresponding intermediate frame image data object. In this regard, the image data padding can be represented by (Cp_left, Rp_top, Cp_right, Rp_bottom) or any other structure or non-structure format that can determine the data padding to be included on each edge. In some embodiments, the data padding for relative edges is evenly distributed (e.g., Cp_left=Cp_right=(Cp / 2), and / or Rp_top=Rp_bottom=(Rp / 2)). Alternatively, in some embodiments, different allocation percentages (e.g., Rp_top=0.75*Rp and Rp_bottom=0.25*Rp) are used to allocate data padding based on the image processing task.
[0133] It should be understood that in the case where the full frame row count is not less than the intermediate frame row count and the full frame column count is not less than the intermediate frame column count, process 1000 may include identifying image data padding that does not include padding. In this regard, in both directions (e.g., in rows and columns), the full frame image size is equal to or greater than the intermediate frame image size, so padding is not required. In some such embodiments, image data padding may be set to a null value or an unreadable value. In other embodiments, image data padding may be set to include the case where all padding values are 0 values (e.g., any combination of (Cp_left, Cp_right, Rp_top, Rp_bottom) is (0, 0, 0, 0) or (Cp, Rp) is directly (0, 0)). Therefore, image data padding will not provide padding, and the generated intermediate frame image data object may only include the cropped image data portion identified in the previous frame.
[0134] In some embodiments, process 1000 includes one or more determination steps regarding whether the captured data should be filled. For example, in at least some embodiments, process 1000 includes determining whether the full frame row count is less than the intermediate frame row count, and / or whether the full frame column count is less than the intermediate frame column count. As described above, the full frame column count can be compared with the intermediate frame column count to determine whether the full frame column count is less than the intermediate frame column count. In some embodiments, the results of the previously performed comparison are utilized to avoid repeated comparisons between the row count and the column count. In other embodiments, an image sensor and / or supporting hardware for performing such comparisons is provided.
[0135] After blocks 1002 and 1004, for example, upon returning to the Fig. 9At block 910, an intermediate-frame image data object corresponding to the captured full-frame image data object may be generated using the identified cropped image data portion and the image data padding. In this regard, the intermediate-frame image data object may be embodied by the cropped image data portion (e.g., where the image data padding does not include padding data). Alternatively, in some embodiments, the intermediate-frame image data object may be embodied by the cropped image data portion, the cropped image data portion having associated padding data based on the image data padding. In a specific exemplary case, where the full-frame image size matches the intermediate-frame image size in both directions, it should be understood that the intermediate-frame image data object may include the same data as the full-frame image data object (which also matches the identified cropped image data portion of the full-frame image data object) because neither cropping nor padding is required.
[0136] in conclusion
[0137] Although exemplary apparatus and processes have been described above, specific implementations of the subject matter and functional operations described herein may be implemented in other types of digital electronic circuitry, or in other types of computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.
[0138] The operations of the processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs, which perform actions by operating on input information / data and generating output, or may be performed by corresponding processing circuits. By way of example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors and any one or more processors of any kind of digital computer. In general, a processor may be specifically configured to execute certain instructions, or to receive instructions and information / data from an accessible memory device.
[0139] Although this specification includes many specific implementation details, these details should not be interpreted as limiting the scope of any disclosed or claimable content, but should be interpreted as a description of specific features for a specific disclosed specific embodiment. Certain features described herein in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a separate embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as acting in certain combinations and even initially claimed, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for a sub-combination or a variation of the sub-combination.
[0140] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or grouped into multiple software products.
[0141] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some specific implementations, multitasking and parallel processing may be advantageous.
Claims
1. A computer-implemented method for intermediate frame image processing, the computer-implemented method comprising: capturing, by a first image sensor, a first full frame image data object associated with a first full frame image size; capturing, by a second image sensor, a second full frame image data object associated with a second full frame image size; outputting a first intermediate frame image data object from the first image sensor and a second intermediate frame image data object from the second image sensor to an image frame multiplexer, the first intermediate frame image data object and the second intermediate frame image data object being associated with an intermediate frame image size, wherein the intermediate frame image size represents a resolution falling between a first resolution defined by the first full frame image size and a second resolution defined by the second full frame image size; selecting, by the image frame multiplexer, a selected intermediate frame image data object including the first intermediate frame image data object or the second intermediate frame image data object; receiving, by a processor, the selected intermediate frame image data object from the image frame multiplexer; and The selected intermediate frame image data object is processed by the processor.
2. The computer-implemented method of claim 1 , wherein outputting the first intermediate frame image data object from the first image sensor to the image frame multiplexer comprises: identifying, by the first image sensor, a size of the intermediate frame image; identifying, by the first image sensor, a cropped image data portion of the first full frame image data object based on the first full frame image size and the intermediate frame image size; identifying, by the first image sensor, a first image data padding based on the first full frame image size and the intermediate frame image size; generating, by the first image sensor, the first intermediate frame image data object based on the cropped image data portion of the first full frame image data object and the first image data padding; as well as The first intermediate frame image data object is transmitted by the first image sensor to the image frame multiplexer.
3. The computer-implemented method of any one of claims 1 to 2, wherein processing, by the processor, the selected intermediate frame image data object comprises processing, by the processor, the selected intermediate frame image data object to determine useful data loss in the selected intermediate frame image data object, the computer-implemented method further comprising: identifying, by the processor, a selected image sensor from the first image sensor or the second image sensor based on the selected intermediate frame image data object; transmitting, by the processor, a full frame request signal to the selected image sensor; receiving, by the processor, a selected full-frame image data object from the selected image sensor, the selected full-frame image data object comprising more data than the selected intermediate-frame image data object; as well as The selected full-frame image data object is processed by the processor.
4. The computer-implemented method according to any one of claims 1 to 2, further comprising: The first image sensor is configured, by the processor, to output the first intermediate frame image data object based on the intermediate frame image size.
5. The computer-implemented method of claim 4, wherein configuring the first image sensor to output the first intermediate frame image data object based on the intermediate frame image size comprises: identifying, by the processor, at least a first full frame image size associated with the first image sensor and a second full frame image size associated with the second image sensor; determining, by the processor, a maximum image size based on the first full frame size and the second full frame image size; determining, by the processor, a minimum image size based on the first full frame image size and the second full frame image size; generating, by the processor, the intermediate frame image size based on the maximum image size and the minimum image size; as well as The first image sensor is configured, by the processor, to output the first intermediate frame image data object based on the intermediate frame image size.
6. The computer-implemented method of any one of claims 1 to 2, wherein selecting, by the image frame multiplexer, the selected intermediate frame image data object comprises: receiving, by the image frame multiplexer, a frame selection signal from the processor, the frame selection signal indicating which intermediate frame image data object among a plurality of intermediate frame image data objects is to be selected; as well as The selected intermediate frame image data object is selected by the image frame multiplexer from among the first intermediate frame image data object and the second intermediate frame image data object based on the frame selection signal.
7. The computer-implemented method of claim 6, further comprising: generating, by the processor, the frame selection signal using a random frame selection algorithm or a sequential frame selection algorithm; as well as The frame selection signal is transmitted by the processor to the image frame multiplexer.
8. The computer-implemented method of claim 6, further comprising: generating, by the processor, the frame selection signal by processing the received selected intermediate frame image data object; as well as The frame selection signal is transmitted by the processor to the image frame multiplexer.
9. A device for intermediate frame image processing, the device comprising at least one processor and at least one memory, the at least one memory comprising computer program code, the at least one memory and the computer program code being configured to utilize the at least one processor to enable the device to perform any one of the methods according to claims 1 to 8.
10. A computer program product, comprising at least one non-transitory computer-readable medium having computer-readable program instructions stored therein, the computer-readable program instructions comprising instructions which, when executed by a device, are configured to cause the device to at least perform any one of the methods according to claims 1 to 8.
Citation Information
Patent Citations
Method and apparatus for synthesizing cartoon video
CN101510314A
Method and apparatus for increasing frame rate of an image stream using at least one higher frame rate image stream
CN104254868A
Image fusion system and method
CN104427245A
Mobile terminal and control method thereof
CN104639735A
Method for reducing parallax of multiple cameras and electronic device supporting same
CN109428997A
