Method and system for color video processing

Abstract

A computer-implemented method (82) for color video processing, the method including the steps of: receiving a CFA image from a CFA imaging device, the CFA image including CFA image pixels defining a plurality of image regions, each image region having at least one pixel of a first color and at least one pixel of a second color, at least one image region representing a portion of an object; receiving a full-color image from the color imaging device, the full-color image representing the portion of the object; storing the full-color image in a memory; and filtering the first color pixels of the CFA image to generate a first color channel including pixels of the first color channel. separating pixels of a first color of the full-color image to generate a second color channel comprising pixels of the second color channel; processing the pixels of the first color channel using a first computer vision algorithm to generate first image metadata; processing the pixels of the second color channel using the first computer vision algorithm to generate second image metadata; and generating processed data using the first image metadata and / or the second image metadata, the processed data comprising a plurality of data points representing the object.

Description

[Background technology]

[0001] When surveying subsea sites, assets, environments, or any other harsh environment installations, it is common to use mobile survey equipment, such as robots or remotely operated vehicles. For simplicity, such facilities are referred to as survey sites and such survey equipment is referred to as mobile equipment.

[0002] The mobile device can include one or more cameras for imaging the survey scene. For computer vision applications, a color camera is typically connected to a processor to receive full-color images and generate metadata (e.g., 3D information).

[0003] The present inventors have devised a new mobile device and method that can enable fast and efficient video processing of color images. Summary of the Invention

[0004] By way of non-limiting overview, embodiments of the present invention relate to a system for efficient color video processing of color filter array (CFA) images, resulting in less data being transferred internally (e.g., from RAM to a processor core) when performing certain tasks, allowing those tasks to be performed faster. Known types of color cameras capture CFA images and demosaic (on-camera) and interpolate them to produce three color channels per pixel across the entire image, tripling the size of the data to produce a full-color image. Thus, a CFA imager or CFA camera outputs a CFA image without further processing the image. For computer vision applications, the full-color image and metadata require a high-bandwidth link for transfer to other applications or visual devices.

[0005] According to a first aspect of the present invention, there is provided a computer-implemented method for color video processing, the method comprising: receiving a CFA image from a CFA imaging device, the CFA image including CFA image pixels defining a plurality of image regions, each image region having at least one pixel of a first color and at least one pixel of a second color; receiving a full-color image from a color imaging device different from the CFA imaging device, the full-color image may include demosaiced image pixels, and the object is imaged by both the CFA imaging device and the color imaging device and is represented in the CFA image and the full-color image; storing the full color image in a memory; separating first color pixels of the CFA image to generate a first color channel comprising pixels of the first color channel; separating pixels of a first color of the full-color image to generate a second color channel comprising pixels of the second color channel; processing the pixels of the first color channel using a first computer vision algorithm to generate first image metadata; processing the pixels of the second color channel using a first computer vision algorithm to generate second image metadata; generating processed data using the first image metadata and / or the second image metadata, the processed data including a plurality of data points representing the object; Includes.

[0006] Thus, the method of the first aspect uses an asymmetric camera system with a CFA imager combined with a full-color imager to perform computer vision processing. The use of both a CFA imager and a color imager provides a synergistic effect in video processing that can improve processing efficiency, allowing computer vision tasks to be performed accurately in real time in harsh environments. The asymmetric system is also well-suited for live data processing, allowing for the generation of 3D models of survey sites in real time in harsh environments without the need for post-processing.

[0007] The method includes processing color video data to generate processed data (e.g., 3D model data) that includes a plurality of data points representing an object. This method of processing color video data is a technical contribution to the field of color video processing that can result in more efficient processing of data, allowing for real-time processing of such data instead of post-processing. This enables real-time decision-making, for example, when an investigation reveals a dangerous situation.

[0008] The method can improve the computational efficiency of the mobile device implementing the method, i.e., reduce the computational load to generate processed data, thereby enabling real-time processing with reduced power consumption, reduced thermal constraints, and fewer and / or less powerful processors, while still being at least as accurate as conventional systems.

[0009] Furthermore, processing high-resolution CFA images and low-resolution color images with the above method can enable nearly double the output resolution, or half the bandwidth, of known systems. Compared to conventional systems, the present method can result in higher frame rates, higher spatial resolution, higher color resolution, and / or full-color regular 2D images for the same processing power.

[0010] The traditional steps of calculating luminance and downsampling images can be partially, or even completely, avoided. This is possible because a single color channel is an adequate proxy for luminance, especially in undersea environments. The majority of tasks in (stereo) video processing and computer vision use luminance. Luminance-chroma representations use brightness and color difference. Luminance is derived from a combination of red, green, and blue (RGB) color channels (or other color channels depending on the camera sensor at the time of acquisition), with the exact formula depending on the end application. The inventors have found that a single color channel can provide a sufficient proxy for luminance for performing video processing and computer vision (e.g., tracking or matching algorithms). Using CFA images means less data is transferred around the system, and full-color images can be demosaiced at a later stage. Therefore, the method can include operating at least one camera in a "RAW" configuration so that the images are not demosaiced. This means that full-color images do not need to be generated until later in the process, if necessary or convenient.

[0011] Full color images, such as those provided by a color camera, are useful for live viewing, and the full color images can be registered to 3D points to generate color 3D models.

[0012] Therefore, image processing and / or computer vision algorithms are performed directly on the CFA image, resulting in less data having to be transferred between processors while maintaining the ability to later restore the full color image if necessary. In constrained bandwidth situations, such as harsh environments, the amount of data flow must be carefully controlled to efficiently utilize the available transmission bandwidth. Running in RAW capture mode allows for a higher capture frame rate and allows for more images to be stored in the cache. When operating in RAW, a high dynamic range can be captured (e.g., 16 bits per pixel) and the image can be bit-shifted to a lower color resolution, such as 8 bits per pixel, to simplify the image for processing. This reduces hardware storage requirements. This contrasts with the traditional demosaiced 8-bit output as YUV 4:2:2 (16 bits per pixel) (subsampled colors), which can be the output of a color imaging device.

[0013] The cameras can be synchronized. If the cameras are calibrated, the relative position of the demosaiced cameras can be determined by pose. If the cameras are not synchronized, the relative position of the demosaiced cameras can be determined by odometer pose and interpolation.

[0014] The step of isolating the pixels does not necessarily isolate and produce a first color channel having all pixels of the first color of the CFA image, but may, for example, include some but not all of the pixels of the first channel of the CFA image.

[0015] The size and / or resolution of each color channel can be varied so that one color channel has a higher resolution than the other.

[0016] The memory may be a volatile or non-volatile memory storage device.

[0017] The image area can include a single basic CFA pattern that includes at least one of each color pixel. For example, if the CFA pattern is a Bayer array, the image area can include at least a square that includes two green pixels, one red pixel, and one blue pixel.

[0018] The CFA imager and color imager can be installed in a stereo system. For stereo processing systems, using full-resolution images is not a requirement. For example, depending on the application, a 4K video frame may have more detail than is needed for tracking. Therefore, only a single color channel from the CFA image needs to be used, and it may not be necessary to perform interpolation of every single color channel pixel to create a full-resolution image.

[0019] When an application requires full resolution, not every frame from the video is needed; for example, some algorithms only require keyframes or other useful infrequent frames.

[0020] The first color channel of the full-color image may be the same as the first color channel of the CFA image. For example, if the first color channel is a green channel, the second color channel is also a green channel. A color channel may be defined as a channel that contains only data representing the intensity of light passing through a specific color filter (e.g., the green (G) color filter in an RGB filter array such as a Bayer filter).

[0021] A CFA image can have a higher dynamic range per pixel than a full-color image. For example, a CFA image can have 16 bits of dynamic range per pixel, while a full-color image can have 8 bits of dynamic range per pixel. This is primarily due to severe bandwidth constraints.

[0022] The method of the first aspect can generate color-processed data by modifying the color of each data point using at least one pixel of a second color in a corresponding image region of the second CFA image, resulting in color-processed data that is substantially identical to color-processed data that can be generated by video processing that is full color but with reduced processing power.

[0023] The method of the first aspect comprises: correlating the or each data point with a corresponding image region of the full color image; generating color-processed data by modifying the color of the or each data point using one or more pixels of the full color image; Includes.

[0024] An advantage associated with processing color video in the first embodiment is that (assuming the same frame rate), a CFA imager can generate lower bandwidth data compared to a full-color imager. This can result in faster processing of the CFA image to generate processed data. However, it may be desirable for the processed data to be colored. Therefore, a full-color image stored in memory can be used to color the processed data. If the color comes from only one camera, color matching is not required when rendering / correcting the color of the or each data point. If two or more cameras capture color information to render / correct the color of the or each data point, color matching would be required. This benefits from faster processing and avoids the need to demosaic the CFA image.

[0025] The first and / or second image metadata comprises at least one of motion, disparity, edges, and segmentation.

[0026] The processed data may comprise at least one of keyframes, selected image segments, and 3D model data.

[0027] The 3D model data can be generated by a 3D reconstruction algorithm.

[0028] If the processed data is keyframes or selected image segmentations, generating the processed image data may be accomplished with only one of the first or second image metadata.

[0029] For underwater 3D surveys, a 3D model of the object / scene can be built incrementally from the 3D model data, periodically adding new points to the model as more images are processed. Full color can be generated when unique data points (e.g., 3D model points) are found, and can be created from the full color imagery or by demosaicing the CFA imagery.

[0030] The second image metadata can be processed using an object detection algorithm to detect objects within the full-color image and assign the objects a label and a location. Object detection algorithms are well suited to processing full-color images.

[0031] The first image metadata can be processed using an object detection algorithm to detect objects in the first image and assign labels and locations to the objects. The first image metadata may be more suitable for object detection if the CFA image is of higher resolution than the full-color image. This is especially true if the object for identification is represented in only a small percentage of the complete (captured) image. The small percentage may be 10%, 5%, 3%, 1%, or 0.5% of the total image area.

[0032] The first color channel can be green. In water, red light is attenuated by the water, and the green channel becomes more dominant. Therefore, a good proxy for luminance can be the green channel. Green pixels can be used in place of luminance in the aforementioned algorithms. In low-light vehicle (e.g., autonomous vehicle) driving conditions, the red channel may be desirable because red in these conditions is a good proxy for luminance.

[0033] The method can further include demosaicing the CFA image to generate a demosaiced full-color image, which can be used to correct the color of each data point for transmission to a thin client or for storage for post-processing.

[0034] The step of generating the first color channel may include a step of interpolating empty pixels from the separated pixels of the first color to generate a full resolution first color channel image of the CFA image, prior to the step of processing the pixels of the first color channel using a first computer vision algorithm to generate first image metadata.

[0035] Empty pixels are pixels that have no associated color information. They can result from ignoring certain colors from a CFA image, and where those colors were ignored, the corresponding pixels in the CFA image have no color information. The process of interpolation generates color information for the empty pixels.

[0036] The step of separating the first color pixels (full resolution pixels) of the full color image may further include the step of downsampling the first color pixels of the full color image to generate a second color channel.

[0037] The color processed data can be transmitted over a low bandwidth link to an external computing device.

[0038] Transmission of full-color high-resolution video from underwater to the surface can be limited by a low-bandwidth tether. Therefore, especially for underwater 3D surveys, it is useful to perform (stereo) processing near the camera and transmit low-bandwidth 3D data (e.g., point cloud and color for each point). This allows live color 3D data to be transmitted to the boat / surface. At the camera, the CFA images can be stored either directly or after being demosaiced as color video when the camera is removed from the water.

[0039] The data points may be 3D model data points. The step of generating processed data includes: processing the first image metadata with a feature detection algorithm to detect a first feature in the CFA image; processing the second image metadata with a feature detection algorithm to detect a second feature within the full color image; generating one of a plurality of data points based on the first feature and the second feature; Includes.

[0040] Feature detection algorithms are well known in the art of computer vision. The first and second features can be the same feature on the object (e.g., a corner of a box). Because both cameras have slightly different but overlapping fields of view, the feature is present separately in both images (CFA and full color), can be identified, and further used to generate 3D model data points.

[0041] The first feature in the CFA image can be tracked with a feature tracking algorithm. Because the CFA image can be low bandwidth compared to a full-color image and the first feature has already been identified, the first feature can be tracked efficiently and quickly using the CFA image. Tracking a feature is computationally more efficient than detecting it again. Therefore, if a feature can be tracked and becomes present, this may occur before full feature detection.

[0042] The 3D model can be used to navigate around / through the environment autonomously and in real time.

[0043] Some or all of the steps of the method of the first aspect may be carried out in a harsh environment, such as a subsea environment.

[0044] According to a second aspect of the present invention, there is provided a method of surveying an environment, the method comprising: Moving a mobile device within a survey site, the mobile device performing the method of the first aspect.

[0045] Thus, video processing is suitable for environmental surveys, where the resolution and frame rate of the multiple CFA images and full color images can be selected based on the stage of the survey process. The first and second resolutions and first and second frame rates of the multiple CFA images and full color images can be selected based on measurements of relative light levels at the survey site.

[0046] The environment may include a subsea environment such as an oil and gas pipeline or rig.

[0047] The investigation task may include at least one of calibration of the CFA imager, feature tracking, feature detection, motion estimation, reconstruction, and disparity estimation.

[0048] Therefore, in computer vision, full color information is not required for tasks such as camera calibration, feature detection, feature tracking, motion estimation, reconstruction, and disparity estimation. All of these tasks, which can be computationally expensive, do not require full color images and typically operate on luminance images. However, full color images can be computationally efficient for object detection algorithms.

[0049] According to a third aspect of the present invention there is provided a mobile device for surveying in harsh environments such as underwater, the mobile device comprising: a CFA imaging device configured to generate a CFA image; a color imaging device configured to generate a full color image, unlike a CFA imaging device; a platform on which the CFA imager and the color imager are mounted with overlapping fields of view; a first data processor and a second data processor, optionally together configured to perform the method of the first aspect; a first data link coupled between the CFA imaging device and the first data processor; a first computer memory coupled to the first data processor; a second data link coupled between the color imaging device and the second data processor; a first casing arranged to be mounted on the platform and defining an enclosed interior space, the first casing including at least a CFA imaging device; First Data Processor and a first data link is housed within the sealed internal space of the first casing. 、C An FA imaging device includes a first casing positioned outwardly from the first casing for capturing a CFA image.

[0050] The mobile device a second casing defining an enclosed interior space and arranged to be mounted on the platform, the second casing including a color imaging device, a second data processor, and a second data link, the color imaging device positioned outwardly from the second casing for capturing full color images; and a third data link coupled between the first data processor and the second data processor, optionally outside the sealed interior spaces of the first casing and the second casing.

[0051] The mobile device may be an underwater camera module for underwater imaging, the environment being a harsh environment and / or a deep sea environment.

[0052] The mobile device may include a remotely operable or autonomous mobile platform such as an underwater remotely operated vehicle (ROV), an autonomous underwater vehicle (AUV), an unmanned aerial vehicle (UAV), an unmanned surface vehicle (UGV), an unmanned underwater vehicle (UUV), or an unmanned surface vehicle (USV).

[0053] In one example, the mobile device may comprise a subsea remotely operable or autonomous vehicle including a propulsion system, a steering system, and a command controller configured to control the propulsion system and the steering system according to command signals provided from a control station or first data processor remote to the vehicle.

[0054] The casings, such as the first and second casings, can be pressure vessels configured to withstand a pressure differential between their exterior and interior, or filled with a pressure compensating gas or fluid, allowing the camera module to be continuously used in harsh underwater environments (such as at least 2 meters of water, or at least 5 meters of water, or at least 10 meters of water, or in some cases at least 20 meters of water, etc.) In other embodiments, the casings can be configured to allow the camera module to be continuously used in other harsh environments (such as in a vacuum).

[0055] The system can store a high-resolution version of the image in a first computer memory for further offline processing while transmitting a lower-resolution version of the image to a remote server. The transmitted image can have lower spatial, temporal, or color resolution. For example, the image can be downsampled to half size, keyframes can be selected, or the color can be reduced from three channels to a single color channel.

[0056] Any feature of any optional aspect may be applied to any other aspect in an analogous manner. [Brief explanation of the drawings]

[0057] [Figure 1] 1 is a system diagram of a mobile device according to one embodiment of the present invention. [Figure 2] 2 is a schematic diagram of a camera module of the mobile device of FIG. 1; [Figure 3] 10 is a system diagram of a mobile device according to a further embodiment of the present invention; [Figure 4] This is a diagram of CFA. [Figure 5] FIG. 1 is a diagram of an isolated green pixel of a CFA according to one embodiment of the present invention. [Figure 6] FIG. 1 is a diagram of the green color channel. [Figure 7a] FIG. 7 is a diagram illustrating a selection of image regions for the green color channel of FIG. 6. [Figure 7b] FIG. 5 shows a selection of corresponding image regions of the CFA of FIG. 4. [Figure 8] FIG. 10 is a diagram illustrating generation of color-processed data from an image region. [Figure 9] 10 is a system diagram of a mobile device according to a further embodiment of the present invention; [Figure 10] 10 is a system diagram of a mobile device according to a further embodiment of the present invention; [Figure 11] 3 is a flowchart of a method for color video processing according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0058] 1 , a mobile device for investigating harsh environments in accordance with one embodiment of the present invention is shown generally at 10. Mobile device 10 is configured for deployment in harsh environments, such as continuous use in excess of two meters underwater, under the sea, or in the vacuum of space, and may comprise a remotely operated vehicle (ROV), an autonomous underwater vehicle (AUV), a robotic platform, or the like.

[0059] Harsh environments impose strict requirements on camera modules, including mechanical stress, radiation stress, chemical reaction, and temperature exposure. Mechanical stress can include pressure, shock, vibration, and coefficient of thermal expansion (CTE) mismatch. Radiation stress can include cosmic rays and magnetic influences from electromagnetism. Chemicals such as salt water, moisture, fuel, and hazardous gases can react with such camera modules. Temperatures below -40°C and / or above 125°C can also be experienced in harsh environments.

[0060] Mobile device 10 comprises a computing device 12 mounted on a remotely operated or autonomous vehicle 14. As described in more detail below with reference to Figure 2, computing device 12 is housed within a protective casing 16 to enable it to operate in harsh environments, such as underwater.

[0061] The computing device 12 includes a data processor 18 communicatively coupled to a first camera 20a and a second camera 20b for receiving images from the cameras 20a and 20b.

[0062] The first camera 20a is a color filter array (CFA) image camera. A CFA is a system in which color filters are used to enable pixels to capture only specific colors in a pattern on an image. CFA imagers (e.g., CFA cameras) are cheaper and lighter than three-sensor cameras and can be more compact than color wheel cameras. CFA imagers are typically single-sensor devices with a color filter array in front of a single sensor to output exactly what is received by the single sensor. CFA images are sometimes referred to as "RAW" images.

[0063] The second camera 20b is a full-color camera or general-purpose camera configured to output a full RGB color image. A CFA imager can have an image processor built in to demosaic the CFA image for output. Demosaicing is the process of generating full-resolution color channels from a CFA image. Full resolution is the resolution of the original (captured) CFA image. Demosaicing involves spatially separating each color pixel into a partial-resolution image / channel. In other words, demosaicing involves separating each color filter pixel onto a partial-resolution color channel. A partial-resolution color channel is an array of only one color filter with empty pixels where other color filters would have been. These partial-resolution color channels can be interpolated to generate a full-resolution color channel for each color (i.e., every pixel is a color filter pixel). A good demosaicing algorithm can hide most of the edge artifacts that can occur during interpolation. A demosaiced CFA image can combine all color channels so that each pixel has three color channels.

[0064] Alternatively, the color imaging device may be a three-sensor device (one per color channel, e.g., red, green, and blue) or a single-sensor device with a color wheel. CFA cameras can also be cheaper compared to color cameras, in part due to reduced on-camera processing power, resulting in fewer components and reduced power consumption.

[0065] The cameras 20a, 20b are mounted in a known spatial arrangement, such as on a common platform 14. The cameras 20a, 20b are arranged with overlapping fields of view such that an object appears in the images output by both cameras but appears differently at different observation angles. The images from the cameras 20a, 20b allow a 3D reconstruction algorithm to scale the object size, which can inform an object detection algorithm when a candidate object is the appropriate size and can create a more accurate 3D model (e.g., a 3D point cloud).

[0066] The 3D reconstruction algorithm can use overlapping images from either of the spatial arrangements of cameras 20 a, 20 b. As a result, a single data point in the 3D point cloud can have multiple pixels (from multiple images) associated with it.

[0067] In other embodiments, one or more additional cameras may be provided with different but overlapping fields of view relative to the first and / or second cameras 20a, 20b or additional camera pairs so that the additional views can be used to reconstruct the object from a different pair of cameras. The 3D reconstruction may be performed in additional data processors, all feeding back into the object detection algorithm.

[0068] The data processor 18 may be coupled to other devices, such as a network interface (not shown), and volatile and non-volatile memory 22. The memory 22 may store images, videos, or metadata, as well as algorithms. The data processor 18 may be suitable for computationally complex image processing and computer vision algorithms. For example, the data processor may include processing cores, including (but not necessarily) GPU cores, and embedded processing for video codecs or AI operations, such as an NVidia® Tegra™ system-on-chip (SoC). The data processor 18 may include one or more communicatively coupled cores and / or separate processors.

[0069] The data processor 18 is configured to execute a 3D reconstruction algorithm to develop a 3D model. To generate a 3D model from video or camera images, the 3D reconstruction algorithm may, for example, identify key points constituting the edges of an object in a second image. The key points can be acquired and associated with corresponding key points from other images (such as previously captured images or images captured simultaneously by other cameras). Knowing the camera pose associated with each image allows rays to be projected from the camera position through the key points in each image and the point where the ray for a particular point intersects or best fits in 3D space representing the 3D location of the corresponding point in the 3D model. The 3D reconstruction algorithm can be structure from motion or any other suitable technique for generating a 3D model from video or camera images. The 3D model can include, for example, a point cloud. The 3D reconstruction algorithm can start building a 3D model from a pair of images or use a second image to extend a 3D model already built by a data processor.

[0070] To create an accurate 3D model, the 3D reconstruction algorithm requires multiple input images. In other words, to model a 3D object, multiple images at different angles around the object help build a more reliable 3D model of the object. If two images are taken consecutively and the capture rate of camera 20a (or camera 20b) is high, the two images are likely to be very similar, and the received latter image adds little information to the previous image. A "keyframe" can be defined by the degree of dissimilarity between two unique views of the object. The 3D reconstruction algorithm can be configured to compare a received third image with a preceding second image (or previous image) to determine the degree of dissimilarity. The third image may contain a different perspective of the object relative to the second image due to movement of the object and / or camera 20a (or camera 20b). If a difference threshold is exceeded, the received image is marked as a keyframe. In other embodiments, keyframes can be identified by a data processor other than data processor 18 and / or by other means, such as waiting a certain amount of time (e.g., 1 second) between keyframes or selecting every nth frame. Alternatively, an inertial measurement unit can be used to determine when the camera position has moved enough to start a keyframe.

[0071] The 3D reconstruction algorithm can process all of the input images, select key frames, and add points that fit well with the model being constructed. The points from the key frames are stitched together to form a point cloud that can be processed into a mesh and rendered as a solid object with computer graphics. Each time a key frame is added, the camera view is typically a unique view on the model. A unique field of view is ideal as input to an object detection algorithm, as a complete set of key frames can provide all views of the object needed to create a complete model of the object. The object detection algorithm can run on the mobile device 10 or remotely, for example, on a server.

[0072] In the illustrated embodiment, computing device 12 further includes a data transceiver 26, which may comprise, for example, a wireless base station.

[0073] Referring now to Figure 2, the illustrated embodiment of a computing device includes a camera module 30 for imaging in harsh environments. The camera module 30 has a casing 16 that defines a waterproof housing having an interior space. One end of the casing 16 includes a transparent window or lens 32. Cameras 20a and 20b are mounted within the interior space and positioned to capture images of the external environment through the window 32. The interior space of the casing 16 can be accessed by removing a casing end cap 16a that is removably coupled to the main body of the casing 16 via an O-ring sealed flange. A seal is provided between the casing end cap 16a and the main body of the casing. A seal may be provided to prevent water ingress. In this embodiment, the casing 16 is formed from stainless steel and is cylindrical in shape to structurally withstand the high pressures that the camera module 30 may be subjected to in harsh underwater environments, such as deep-sea environments and / or subsea oil and gas infrastructure sites. In other embodiments, the material and / or shape of the casing 16 can be modified to provide stress, chemical, and / or temperature resistance depending on the deployment environment (e.g., aluminum, beryllium copper, titanium, plastic, ionomer, PEKK, carbon fiber, or ceramic). The material and shape of the casing 16 preferably provide a strong, rigid structure. The shape of the casing 16 can be any pressure-resistant shape, such as a prismatic or cylindrical shape. The cameras 20a and 20b can form part of the casing 16 such that the casing 16 forms a fluid seal with at least a portion of the cameras 20a and 20b (e.g., the lenses of the cameras 20a and / or 20b). Other end cap seals can be formed using methods including through bolts or threaded tubes. Some or all of the components of computing device 12 may be housed within casing 16. Ports P1-P5 may be provided for wired connections to other camera modules, camera modules, etc. to provide external data links while the camera modules are operating in harsh environments such as subsea.

[0074] While in the above-described embodiment both cameras 20a, 20b are housed within a common casing 16, in other embodiments each camera may be mounted within its own casing to form a respective camera module 12a, 12b. For example, as shown in FIG. 3, a computing device 10a according to a further embodiment may include a pair of camera modules 12a and 12b mounted on a remotely operated or autonomous vehicle 14a. Alternatively, computing device 10a may be mounted on ..., 12b mounted on a remotely operated or autonomous vehicle 14a. 12The camera modules 12a and 12b may be functionally identical to each other. The camera modules 12a and 12b have protective casings 16a and 16b for operation in harsh environments, such as underwater. The camera module 12a includes a data processor 18a communicatively coupled to the camera 20a to receive images from the camera 20a. The camera 20a is a color filter array (CFA) image camera. The camera module 12b includes a data processor 18b communicatively coupled to the camera 20b to receive images from the camera 20b. The camera 20b may be a full-color camera configured to output full RGB color images or a general-purpose camera. Either the camera modules 12a and 12b or both may include the transceiver 26.

[0075] Referring to Figure 4, an example of a CFA pattern (output of CFA imager 20a) is shown, namely, a Bayer array 40 using green G, red R, and blue B pixels. In other embodiments, any CFA pattern and color can be used. Bayer array 40 is a filter placed in front of a single sensor camera, and when an image is captured, an image is produced consisting of green G, red R, and blue B pixels. Each pixel has an associated numerical value that represents the amount of light of each of that color in the image subject.

[0076] Known computer vision tasks and algorithms use luminance as an input to generate an output. Luminance is derived from all color combinations in a CFA pattern, and typically each value associated with each color is weighted differently to produce a luminance value.

[0077] In most single sensor cameras that use a CFA (e.g., color camera 20b), the raw data from the captured CFA image is immediately demosaiced to produce three color channels, each with the same resolution as the CFA. Demosaicing is a two-step process: i) creating the same color pixel for each color (e.g., red, green, and blue) of the CFA; This involves two steps: (i) separating or splitting the colors of the CFA pattern to generate a pattern of cells (pixel pattern 50, as shown in Figure 5); and (ii) interpolating each pattern of same-color pixels to generate three color channels (e.g., red, green, and blue), each with the same resolution as the CFA. These three color channels can be combined to generate a full-color image. This process is performed automatically for each image from a conventional CFA color camera by a camera-embedded processor.

[0078] The inventors recognized that a single color in a CFA can be used as a proxy for brightness without significantly impacting the computer vision task or algorithm. The single color can be the most weighted color, or the color that is most present in a given image environment. For example, underwater environments have a large proportion of green light compared to other colors of light. Also, green light is typically weighted the most for brightness calculations. Therefore, in underwater environments, it may be advantageous to perform computer vision tasks and algorithms with only green input.

[0079] Figure 5 shows that, in contrast to the conventional method described above, only one color channel is required, and therefore raw data from a captured CFA image is transferred from the camera to the data processor 18. Pixels of the first color are isolated, and pixels of the remaining colors are removed, generating a pixel pattern 50. Where they are absent, empty pixels V are shown along with existing pixels of the first color G. This pixel pattern 50 represents a subsampled green image, referred to herein as a color channel. This pixel pattern can be directly processed by computer vision algorithms.

[0080] The empty pixels V can be filled to generate a full-resolution color channel image containing the pixels of the color channels, as shown in Figure 6. Assuming a Bayer-type CFA, the four methods for generating a full-resolution color channel image are as follows: 1. Simply interpolate the missing color pixels from their four neighbors. 2. Demosaicing only the color channels. 3. Simply average the two color pixels in a 2x2 block (this produces a smoothed image at quarter resolution). 4. Select only pixels where both coordinates are even (this produces a decimation of 1 / 4 resolution).

[0081] These methods can be adapted to other types of color filter arrays, such as Fuji X-trans, RGBW, or RGBE. For method 4, known filtering techniques can be implemented to avoid aliasing problems.

[0082] For example, in Figure 5, empty pixel V1 can be filled by interpolating its four neighbors G1, G2, G3, G4. The interpolation can be repeated for all empty pixels V in the image.

[0083] 6 shows the resulting full-resolution color channel image 60, which may be useful for several processes, such as detecting errors in the final processed data, improving object detection and 3D reconstruction algorithms (i.e., machine learning), and displaying information.

[0084] The color channel 50 can be processed using a first computer vision algorithm to generate first image metadata.

[0085] Image metadata can be data defining shapes or object edges present in the imaged environment. Alternatively, the metadata can define disparity, segmentation, or motion. Image metadata corresponding to motion can be found from two consecutively captured CFA images, both of which can be processed to generate a first color channel, both of which can be processed to generate first image metadata, and the two images can be processed to find areas where motion occurred and recorded as image metadata.

[0086] Image metadata can also be generated from a full-color image. This full-color image can be generated from a color camera or CFA camera with demosaicing capabilities. This full-color image can be used to separate (split) a full-resolution color channel from another full-resolution color channel, which can be a full-resolution green color channel. For efficient processing, the full-color image is typically subsampled to generate a subsampled-resolution green color channel that is processed using a computer vision algorithm. Thus, the full-resolution color channel or subsamples can be used as a substitute for luminance and downsampled / subsampled for processing by a first computer vision algorithm to generate second image metadata.

[0087] Once the image metadata is generated, it can optionally be further processed using an object detection algorithm. The algorithm can detect objects within the processed images and further assign labels and / or locations to the objects. For example, an object may be completely depicted in a single image (e.g., a handle), or the object may be partially depicted across multiple images (e.g., a pipeline). The object detection algorithm is optionally run separately on full-color images, e.g., from a full-color camera. Because color is an important factor in helping to identify objects, inputting full-color images into the algorithm results in superior object detection in terms of accuracy and speed. Object detection consists of two processes: feature detection and object classification. Feature detection and object classification can be performed using algorithms known to those skilled in the art, although object classification is potentially more resource-intensive (e.g., power, time, etc.).

[0088] Object detection and 3D reconstruction can be performed at different speeds. For example, 3D reconstruction can be updated frequently to generate a high-resolution 3D model, while object detection can be performed when new objects are present in the environment. When new objects are not present, feature detection and tracking algorithms can be performed without an object classification process. These feature detection and tracking algorithms can be accurately performed using only the color channels, i.e., without the need for full-color images. Tracking allows the output of the object classification process, i.e., object label and / or location, to be updated without the need to perform object classification. Tracking algorithms perform efficiently when the difference between two images is small. This can be achieved with high frame rates or slow-moving objects or cameras. In one example, feature detection is always performed on the color channels, and an object classification algorithm is performed using the location of features on the full-color image only if the feature detection algorithm does not recognize a set of features representing a new object. When the color channels are derived from the full-color image, feature locations can be found efficiently because the features are identical.

[0089] The first image metadata and the second image metadata are processed to generate processed data. The processed data includes a plurality of data points. The plurality of data points may constitute 3D model data. The image metadata is processed through a 3D reconstruction algorithm that may generate a point cloud. Alternatively, the 3D reconstruction algorithm may generate a mesh model. or any other type of model can be generated. The points in the point cloud are data points. The data points correspond to particular pixels, or alternatively, clusters of pixels, in the original image from which they were derived, such that each point can have an associated color value.

[0090] Alternatively, the processed data can be keyframes, or selected image segments, or a combination of keyframes, selected image segments, and / or 3D model data. If the processed data is a keyframe, the plurality of data points are simply pixels of the image. If the image data is selected image segments, it can identify objects in the image, specifically objects in the field of view when the image was captured. In this case, the plurality of data points can be pixels that represent or make up the image, or vice versa.

[0091] Each data point is further correlated with a corresponding image region in the CFA image and / or the full-color image. If the data point is a data point of a 3D model, the data point is derived from both the first image metadata and the second image metadata. Thus, the color of the data point can be derived using either the first image metadata or the captured image associated with the second image metadata.

[0092] 7a and 7b, in an example of generating the color of a data point from a CFA image, data point G5 is correlated with a corresponding image region IRG in first color channel 60 and a corresponding image region IRC in CFA image 40. In this embodiment, image region IRG, IRC consists of nine pixels, although in other embodiments the image regions may be larger or smaller.

[0093] The color or colors of the data points can be extracted from the relative position of the image regions in a full color image captured from a color camera.

[0094] The full color image is stored in memory prior to or in parallel with separating the color channels from the full color image.

[0095] Alternatively, one or more colors of a data point can be extracted from its relative position in the image region of the CFA image. In this alternative embodiment, each full-color pixel can be generated by combining corresponding pixels from each color channel. For example, FIG. 8 illustrates the generation of full-color pixel C from an image region of a CFA image. A corresponding color pixel from image region IRC is extracted from image region IRC, as indicated by blue pixel IRB in image region IRC and red pixel IRR in image region IRC. The red pixel IRR is interpolated to generate a central red pixel R1 in image region IRC. A full-color pixel C corresponding to pixel G5 is then generated from the corresponding color pixels in image regions IRR, IRB, and IRG. This is done using green pixel G5 in image region IRG, blue pixel B1 in IRB, and central red pixel R1. Each color pixel G5, B1, and R1 corresponding to full-color pixel C is weighted by a corresponding numerical value representing the amount of light of that color. The corresponding numerical value can be coded from 255 (i.e., one byte). This is essentially a demosaicing of the image region IRC of the CFA image.

[0096] Alternatively, both the first and second image metadata can be used if some statistical averaging or weighting is used. This is advantageous because multiple pixels may not have the same exact color due to camera angle and shadows. Deriving the color of the 3D model data points is most efficient (power, time) when full color imagery is used. The amount of power used is low, especially in subsea environments due to sealed casings. This is crucial in any harsh environment, and for the processing to occur in real time, it must keep up with the frame rate of the captured images. However, the frame rate can be changed to allow the processing to occur in real time. Alternatively, using both images will give the best accuracy, and processing power and time are not an issue.

[0097] Alternatively, the 3D model data need not be colored, for example, for autonomous rover control through or exploration of an environment.

[0098] FIG. 9 illustrates processes within a data processor 18 occurring on a computing device such as 12 or 12a and their relationship to memory 22 of FIGS. 1 and 2, shown as storage device 22a and optional cache 22b in FIG. 9.

[0099] Storage device 22a stores CFA images from camera 20a and / or full-color images from camera 20b. The stored CFA images are raw data. Full-color video can be reconstructed offline from the raw data via demosaicing after storage. Alternatively, the CFA images can be demosaiced 70 to generate full-color images before being stored in storage device 22a. Storage device 22a can be connected via a local high-bandwidth link.

[0100] The raw data is processed at 72 to isolate pixels of a first color, producing a first color channel 50. The full color image is also processed at 72 to isolate pixels of the first color, producing a second color channel, which may be at full resolution or sub-sample resolution.

[0101] The video processor 74 generates first image metadata from the first color channel 50 using a computer vision algorithm. The computer vision algorithm may process one or more frames at a time to generate the first image metadata. The video processor 74 may also generate second image metadata from the second color channel using a computer vision algorithm. The computer vision algorithm may process one or more frames at a time to generate the second image metadata.

[0102] Alternatively, video processor 74 may identify particular portions of an image (e.g., objects within an image) using image metadata that may be transmitted via transceiver 26. The transceiver link is a low bandwidth link.

[0103] The video processor 74d executes a 3D reconstruction algorithm, such as simultaneous localization and mapping (SLAM), to generate a plurality of data points that constitute a 3D point cloud. The 3D reconstruction algorithm takes as input at least the first and second image metadata. This 3D point cloud can be used to view the model outside the field and to assist object detection algorithms to autonomously navigate the autonomous vehicle 14. Note that in this embodiment, there is no stage where all frames are demosaiced. The 3D point cloud is optionally Transceiver It can be sent via 26.

[0104] Alternatively, video processor 74d may generate or select key frames, selected image segments of interest based on the first and / or second image metadata.

[0105] In an optional step, the colorizer 76 receives multiple data points and generates colored data points from full-color pixels or sets of pixels of the full-color image stored in the storage device 22a. The colorizer can extract specific color pixels needed for the colored data points, or it can extract the entire image and then select the specific color pixels needed for the colored data points. This may depend on how resource-constrained each processor 18a, 18b is at a particular time. This generates a full-color 3D point cloud that can be transmitted via the transceiver 26. This full-color point cloud can be used to autonomously navigate the autonomous vehicle 14 and view models outside the field. Note that in this embodiment, at no stage are all frames from the CFA imager demosaiced.

[0106] Alternatively, the colorizer 76 can receive the corresponding full-color image from the storage device 22a by demosaicing the stored raw data in a second thread / processor. Additionally, the raw data in the storage device 22a can be transmitted via the transceiver 26 for off-site processing as indicated by the link 78.

[0107] In other embodiments, the raw data may be processed at 72 to interpolate pixels of a first color. These interpolated pixels form a full-resolution first color channel, such as 60 in FIG. 6, and are stored in cache 22b. Cache 22b may store many of these images. These images may be used to assist object detection algorithms or for off-site viewing.

[0108] Referring to FIG. 10, a further embodiment illustrates a process for generating processed video data using two data processors 18a and 18b (although these may be a single processor 18) occurring on computing devices 12a and 12b, respectively, and their relationship to memory 22 (shown as storage devices 22a, 22c). Specifically, FIG. 10 illustrates the generation of 3D model data and the transmission of said data. However, the 3D model data can be used for navigation as well as surveying harsh environments. Common elements in FIG. 9 are labeled as such and operate in substantially the same manner unless otherwise specified.

[0109] The storage device 22c stores the full-color images from the camera 20b, which may optionally be used by the colorizer 76.

[0110] The full color image is processed at 72b to separate the green color channel from the full color image, producing a color channel similar to full resolution color channel image 60 of Figure 6. The color channel is provided to downsampler 72c, which downsamples the color channel to produce a second color channel, e.g., a subsampled green image similar to pixel pattern 50 of Figure 5.

[0111] The video processor 74b generates first image metadata from the first color channel using a computer vision algorithm, specifically, a feature detection and tracking algorithm, which may process one or more frames at a time to generate the first image metadata.

[0112] The video processor 74c generates second image metadata from the second color channel using a computer vision algorithm, specifically, feature detection only. No. 2 It can process one or more frames at a time to generate image metadata.

[0113] Optionally, colorizer 76 receives a plurality of data points and generates colored data points using a full color image stored in storage device 22c.

[0114] Therefore, the computer vision processing pipeline can be advantageously separated from the visualization pipeline, meaning that processors and / or cores can be allocated according to the processing needs of each pipeline. The advantage of this separation is that the data (e.g., pixels of a full-color image and second image metadata) transferred from 18b to 18a has a relatively low bandwidth. This makes it possible to provide a low-bandwidth link between the two data processors 18a and 18b, which is advantageous in harsh environments such as subsea.

[0115] CFA cameras allow for the efficient generation of a single color channel (i.e., the first color channel 50) at raw resolution. This single color channel at raw resolution can be used for feature tracking and feature detection. This color channel does not need to be interpolated. Considering processing bandwidth limitations, processing a single color channel at raw resolution allows for a greater frequency of images to be processed per unit time because the images have a relatively lower resolution than full-color images. Full-color images allow for efficient execution of object detection algorithms, although this process is resource-intensive and takes longer than feature tracking and feature detection. Feature detection algorithms can be performed from a second color channel derived from the full-color image. Feature detection on the second color channel is necessary to contribute to the generation of a 3D model. However, both object detection and feature detection are resource-intensive, leaving little processing power for processing bandwidth limitations. Therefore, full-color images can have a lower spatial or temporal resolution than CFA cameras to keep up with real-time use of CFA cameras. In other words, CFA cameras have the processing headroom to implement alternative modes.

[0116] For example, a CFA camera can operate at a higher frame rate than a color camera, but the color and CFA cameras can operate at the same spatial resolution. This can result in better tracking due to less motion between frames. Alternatively, a CFA camera can operate with a higher dynamic range to improve tracking in low-light conditions.

[0117] In another example, a color camera can operate at a lower frame rate than a CFA camera, but at a higher resolution.

[0118] In another example, a full-color camera can operate at the same frame rate as a CFA camera, but at a lower spatial resolution, e.g., half. This allows for full 1:1 synchronization between the CFA and full-color cameras, resulting in a less expensive full-color camera. Another advantage is that the higher-resolution CFA images can provide better-quality frames that can aid in object detection when color images are not of high enough quality for the object classification process. Objects of interest are known to be very small, covering only a certain number of pixels, e.g., less than 5000 pixels, 2000 pixels, 1000 pixels, 500 pixels, or 250 pixels.

[0119] The high resolution images from a CFA camera combined with the high frame rate and lower resolution of a full image camera can nearly double the output resolution at half the bandwidth of typical prior art color video processing systems.

[0120] Both the CFA and full-color cameras can be synchronized. If the full-color camera has a lower time resolution than the CFA camera, this is due to a lower frame rate. This could mean that every other image is synchronized, or if the camera is at a very high frame rate frequency, such as 300Hz, adjacent frames may not be synchronized in some cases due to small deviations. This can be used for other use cases where a high frame rate is required.

[0121] Referring to FIG. 11, embodiments of the present invention extend to a computer-implemented method, generally designated 82, for color video processing.

[0122] In step 90, the method includes receiving a CFA image from a CFA imaging device, the CFA image including CFA image pixels defining a plurality of image regions, each image region having at least one pixel of a first color and at least one pixel of a second color.

[0123] In step 91, the method includes receiving a full-color image from a color imaging device, the full-color image may include demosaiced image pixels.

[0124] The cameras are positioned so that the object is imaged by both the CFA imager and the color imager and is represented on the CFA image and the full color image.

[0125] In step 92, the method includes storing the full color image in memory.

[0126] In step 93, the method includes separating pixels of a first color of the CFA image to generate a first color channel comprising pixels of the first color channel.

[0127] In step 94, the method includes separating pixels of the first color of the full color image to generate a second color channel comprising pixels of the second color channel.

[0128] In step 95, the method includes processing the pixels of the first color channel using a first computer vision algorithm to generate first image metadata.

[0129] In step 96, the method includes processing the pixels of the second color channel using a first computer vision algorithm to generate second image metadata.

[0130] In step 98, the method includes generating processed data using the first image metadata and / or the second image metadata, the processed data including a plurality of data points representing the object.

[0131] Mobile devices and methods according to embodiments of the present invention can be utilized in offshore and subsea surveying, but can also be applied to any surveying system for hazardous environments, such as space, nuclear, or mining. Surveys can be performed autonomously, semi-autonomously, manually via remote control, or manually with a minimal crew size to reduce risk. Sending a crew to a dangerous location involves the inconvenience of taking safety precautions and therefore increases costs. The mobile device can be, for example, an ROV or AUV.

[0132] Although the invention has been described above with reference to one or more preferred embodiments, it will be understood that various changes or modifications can be made without departing from the scope of the invention as defined in the appended claims. The term "comprising" can mean "including" or "consisting of", and therefore does not exclude the presence of elements or steps other than those listed in any claim or the specification as a whole. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. 1. A computer-implemented method for color video processing, comprising: receiving a CFA image from a CFA imager, the CFA image including CFA image pixels defining a plurality of image regions, each image region having at least one pixel of a first color and at least one pixel of a second color, at least one image region representing a portion of an object; receiving a full color image from a color imaging device, the full color image representing the portion of the object; storing the full color image in a memory; separating the first color pixels of the CFA image to generate a first color channel comprising pixels of the first color channel; separating the first color pixels of the full color image to generate a second color channel comprising pixels of a second color channel; processing the pixels of the first color channel using a first computer vision algorithm to generate first image metadata; processing the pixels of the second color channel using the first computer vision algorithm to generate second image metadata; generating processed data using the first image metadata and / or the second image metadata, the processed data including a plurality of data points representing the object. Computer-implemented methods.

2. correlating the data points with corresponding image regions of the full color image; generating color-processed data by modifying the color of the data point using at least one pixel of the full-color image; The method of claim 1 further comprising:

3. The method of claim 1 , wherein the first and / or second image metadata comprises at least one of motion, disparity, edges, and segmentation.

4. The processed data includes keyframes, selected image segments, and 3D model data. The method according to claim 1 or 2, comprising at least one of:

5. The method of claim 1 , wherein the second image metadata is processed with an object detection algorithm to detect the objects in the full-color image and assign labels and locations to the objects.

6. 6. The method of claim 1, wherein the first image metadata is processed with an object detection algorithm to detect the object in the first image and assign a label and a location to the object.

7. The method of claim 1 , wherein the first color channel is green.

8. 8. The method of claim 1, wherein separating the pixels of the first color of the full-color image further comprises downsampling the pixels of the first color of the full-color image to generate the second color channel.

9. 9. The method of claim 1, further comprising: demosaicing the CFA image to generate a demosaiced full-color image; and optionally storing the demosaiced full-color image in the memory.

10. The method of claim 1 , further comprising transmitting the processed data to an external computing device over a low bandwidth link.

11. the data points are 3D model data points, and the step of generating processed data comprises: processing the first image metadata with a feature tracking algorithm to track a first feature in the CFA image; processing the first image metadata with a feature detection algorithm to detect the first feature in the CFA image; processing the second image metadata with a feature detection algorithm to detect second features within the full-color image; The method of claim 4 , further comprising: generating one of the plurality of data points based on the first feature and the second feature.

12. 1. A method for surveying an environment, comprising: A method comprising the step of moving a mobile device within a survey site, said mobile device implementing a method according to any one of claims 1 to 11.

13. 1. A mobile device for surveying an environment, comprising: a CFA imager configured to generate a CFA image; a color imaging device configured to generate a full color image, different from the CFA imaging device; a platform on which the CFA imager and the color imager are mounted with overlapping fields of view; a first data processor and a second data processor together configured to carry out the method of any one of claims 1 to 12; a first data link coupled between the CFA imaging device and the first data processor; a first computer memory coupled to the first data processor; a second data link coupled between the color imaging device and the second data processor; a first casing arranged to be attached to the platform and defining an enclosed interior space, wherein at least the CFA imaging device, the first data processor and the first data link are housed within the enclosed interior space of the first casing, and the CFA imaging device is arranged to face outward from the first casing for capturing CFA images.

14. a second casing defining an enclosed interior space, the second casing being arranged to be mounted to the platform and comprising the color imaging device, the second data processor, and the second data link, the color imaging device being arranged facing outwardly from the second casing for capturing full color images; a third data link between the first data processor and the second data processor, optionally coupled outside the sealed interior spaces of the first casing and the second casing; The mobile device of claim 13 further comprising:

15. 15. The mobile device of claim 13 or 14, wherein the mobile device is a subsea mobile device.

Images