Protocol for communication between platforms and imaging devices

Abstract

According to some embodiments, a protocol allows communication between platforms and imaging devices. This enables, for example, the platform to specify particular types of information it might want, the format of information it might prefer, and other information that can reduce the amount of processing required on the platform. For example, in gesture recognition software, the platform conventionally receives a continuous video stream that needs to be parsed, searched, and processed to identify gestures. This can consume communication bandwidth between platforms and imaging devices, especially in cases where wireless or other bandwidth-limited communication methods may be involved.

Description

General state of the art

[0001] This generally applies to computer-controlled devices, including image processing device peripherals such as printers, monitors or displays, and cameras.

[0002] Conventional computing platforms such as laptops, desktop computers, and tablets, to name a few examples, can connect to and receive information from image processing devices. As used herein, an "image processing device" is anything that can produce or display an image, including a monitor, display, camera, image sensor, printer, or fax machine.

[0003] Conventionally, the platform simply receives raw data from the image processing device and then performs the necessary processing of the raw data.

[0004] US 2009 / 0054102 A1 describes a method for determining a preferred image format in a user device (UE) that supports mobile video calling between UEs. Each UE should have a camera and a display when receiving video transmission control information from another UE that includes a preferred image format. If the preferred image format requested by the other UE is acceptable, video transmission control information is sent to the other UE containing at least one acceptable response message and an acceptable new preferred image format, depending on the acceptance of the preferred image format requested by the other UE.

[0005] DE 10 2006 038 155 A1 describes a method for medical image processing in a distributed network, the method comprising: communication on a distributed network having a server, a client and a communication path with bandwidth, wherein the distributed network has system resources related to the server, the client and the communication path; monitoring of the system resources and the bandwidth with at least one process monitoring device to generate monitoring data, and recommendation of an allocation of at least some of the system resources for processing three-dimensional image data in order to generate two-dimensional image data that can be displayed on the client, wherein the allocation is based at least partly on the monitoring data.

[0006] US 2010 / 0231970 A1 describes a printing system comprising a sending terminal that transmits printable content and a printer terminal that receives and prints the content, the terminals being connected via an IP network.

[0007] US 7,868,912 B2 describes a video surveillance system that extracts video elements and the occurrence of events from those video elements using event discriminators. The system can trigger a response, such as an alarm, based on the extracted events. Summary of the invention

[0008] The problem underlying the invention is solved by the subject matter of the independent claims. Further advantageous embodiments are specified in the dependent claims. Brief description of the drawings

[0009] Some embodiments are described with reference to the following figures: Fig.1 is a block diagram of an embodiment of the present invention; Fig. 2 is a flowchart of an embodiment of the present invention; and Fig. Figure 3 is a flowchart for a further embodiment of the present invention. Detailed description

[0010] According to some embodiments, a protocol allows communication between platforms and imaging devices. This enables, for example, the platform to specify particular types of information it might want, the format of information it might prefer, and other information that can reduce the amount of processing required on the platform. Therefore, a protocol can provide standardized procedures for control and status information to be passed between devices, such as control messages to maintain device functions and modify processing characteristics and behavior, and status messages to indicate available device features and options. For example, in gesture recognition software, the platform conventionally receives a continuous video stream that is to be parsed, searched, and processed to identify gestures.This can consume communication bandwidth between platforms and image processing devices, especially in cases involving wireless or other bandwidth-limited communication. Therefore, in some embodiments, it is advantageous to enable communication between the platform and the image processing device to specify the information requested by the platform, for example, to reduce the need to send unnecessary data that would simply be discarded.

[0011] Similarly, any image sink, such as a display, monitor, printer, or fax machine, can specify to the image source, such as a computer system or platform, the format in which it wishes to receive the data. For example, a display or printer requiring specific file types, particularly data densities or special protocols using specific borders, can specify this information to the source. The source can then perform the processing to provide the information to the image sink. Furthermore, a protocol embodiment of this invention can include actual cameras, printers, and displays, as well as processing units to act on behalf of the cameras and displays, and virtual cameras, printers, and displays operating within another device.For example, a laptop computer can run a virtual camera that can communicate with a real printer using this protocol, with the virtual camera acting on behalf of the non-smart laptop camera and providing a smart wrapper to implement this protocol.

[0012] Similarly, in some embodiments, image sources can specify the properties of image data they can provide, offer alternatives for selecting other formats, and receive feedback from the sinking device about the preferred format.

[0013] Imaging devices can include displays, printers, image processors, image sensors, and fax machines, to name a few examples. In some embodiments, these peripherals possess sufficient intelligence to perform data analysis and manipulation, receive commands, and communicate responses to those commands. Therefore, these peripherals will generally be processor-based systems that also include memory or storage.

[0014] Possible applications include face recognition, object recognition, motif recognition, perception-oriented information about light sources and direction vectors in a motif, and colorimetric properties of source and target devices.

[0015] As an initial example, an image sensor in a camera might contain intelligence to modify the type of processing performed or the corresponding metadata generated to meet the needs of the consuming endpoint platform. For instance, some platforms might require processed image metadata in the form of metrics such as point of interest coordinates, object coordinates, object counts, and other descriptive information, either with or without video or other image information. Another image processing device, such as a printer, might not require any metadata and could simply request raw video processed in a specific way.

[0016] As another example, a smart camera can be instructed to search for faces with specific attributes. A smart printer can tell a camera to provide raw image data that fits into the printer's color space device model for optimal print rendering, while intelligent application software can be allowed to request the camera to prepare a three-dimensional depth map of a subject at ten frames per second. Similarly, a printer or display can request the camera to provide the locations and regions of an area of ​​objects, such as surfaces, so that the printer or display can perform intelligent extensions of the objects, such as facial regions, in an optimized way to achieve the best viewing results.Therefore, an image capture device may be able to recognize a wide variety of objects and communicate information about the objects to a printer or display using a standardized protocol of the type described here, thus enabling the printer, display or other playback device to optimize rendering.

[0017] Therefore, in some embodiments, a standardized bidirectional protocol can be implemented to enable communication of imaging data specifications between the platform and peripheral device. In some embodiments, this can result in more efficient data transmission and a reduction in the transmission of unnecessary data.

[0018] The protocol can be implemented at a high level as Extensible Markup Language (XML), American Standard Code for Information Exchange (ASCII) text command streams sent bidirectionally between image processing devices via existing standard hardware protocols used by cameras, including but not limited to Universal Serial Bus (USB), Mobile Industry Processor Interface (MIPI) (specifications available from MIPI Alliance, Inc.), Peripheral Components International Express (PCI), 3.05 specification (PCI Special Interest Group, Beaverton, OR 97006, October 8, 2010), or the protocol can be an extension of existing hardware protocols. Alternatively, a new hardware protocol can be developed as a bidirectional channel, such as in MIPI, USB, PCIE, or even using video standards like H.265 (High Efficiency Video Coding, February 2012, available from the Fraunhofer Heinrich Hertz Institute) and CODEC formats.See H.265 available from ISO / IEC of the Moving Pictures Experts Group (MPEG).

[0019] Use cases include printers with a smart protocol that instruct a camera with a smart protocol on how to process images so that they are printed optimally for each device's given color gamut. Another use case is a camera with a smart protocol that can be configured to generate information only when a specific face is detected. This allows facial details to be sent to the smart camera device along with appropriate face-matching coordinates and confidence levels, which can be sent to the platform with or without the corresponding image.This exchange can involve sending standardized sets of interest or descriptor sets to search for, such as "search for faces with these characteristics, search for the following gestures, search for the following objects and report only if the object has been found", and then sending the coordinates, descriptor information, confidence level and an entire image frame or frame portion containing the object.

[0020] Other use cases include smart protocol displays that can send their colorimetrically accurate color gamut map and device color model to a camera, enabling the camera to produce optimal images for that display. Another application involves facial tracking software that uses a communication protocol to send commands to a smart protocol camera sensor chip, requesting only coordinates to facial rectangles along with corresponding points of interest and other image description details. As an additional application, a 3D printer can use a communication protocol and channel to send configuration commands to a smart 3D camera.These configuration commands can include special commands or commands for various three-dimensional (3D) sensor technologies, including, but not limited to, stereo, time-of-flight (TOF) particle, structured light, and the like. A 3D printer then requests only a depth map and a set of triangles in a 3D triangle depth mesh, as well as structures on each triangle in the 3D triangle depth mesh, from the image sensor camera device with corresponding colors for each polygon. This allows a three-dimensional model to be printed directly from the camera's depth map and color information on the 3D printer or the 3D triangle depth mesh. The same 3D triangle depth mesh can also be provided to a 3D display via a standardized protocol of the type described here to enable full 3D rendering on the display.

[0021] Therefore, with reference to Fig.1. A computer system 10 includes a platform 12 with memory 16, a processor 14, and an interface 18. The interface 18 can connect to image processing devices such as a camera 20, a printer 22, and a monitor 24. Each of the devices 20, 22, and 24 can be hardware devices with hardware processors 50 and internal memory 52. ​​A face recognition application 26 and a gesture recognition application 28 can also be stored in the memory 16. A protocol of the type described here allows devices to program each other to perform special functions. For example, a smart printer can send program source code or executable code to a smart camera to perform special processing on behalf of the printer.

[0022] Interface 18 can implement one of a variety of different interfaces, including MIPI, USB, Unified Extensible Firmware Interface (UEFI) (UEFI specification, version 2.3.1, April 18, 2011), Hypertext Markup Language (HTML), or even Transmission Control Protocol / Internet Protocol (TCP / IP) sockets or Uniform Data Protocol (UDP) datagrams. Other communication channels include both wired and wireless networks. The interface can implement a protocol, which can be a request / response protocol, a polled protocol, or an event or interrupt event protocol, to name a few examples. The protocol can also use a common command and status register (CSR) memory or register interface, or a stream protocol interface such as HTTP, datagrams in a socket over TCP / IP, to name a few examples.Each protocol method can be used in conjunction with some embodiments of the present invention.

[0023] In the Fig. 2 and Fig. 3 sequences shown, which contain the protocol source sequence 30 in Fig. 2 and the protocol sink sequence 40 shown in Fig. 3. These elements can be implemented in software, firmware, and / or hardware. In software and firmware implementations, the sequences can be implemented by one or more non-volatile, computer-readable media that store computer-executable instructions. In some embodiments, the non-volatile, computer-readable media can be optical, magnetic, and / or semiconductor storage media.

[0024] With reference to Fig.Block 2 begins the protocol source sequence 30 for requesting data by receiving a request, as shown in block 32. The request can be received by a platform or a peripheral device such as a camera. The request can be translated into suitable commands that are useful within the video receiving / requesting device, as shown in block 34. Then, the raw data that can be received can be filtered, as shown in block 36, to conform to the format specified in the request. Therefore, the format can include various data formats, different data sizes, specifications of special objects to be located in data and images, location of specific text elements, or any other requests. The filtered data is then sent to the sink, as shown in block 38. In some embodiments, the sink can be the receiving device, such as a monitor or the platform.

[0025] Fig. Figure 3 shows the protocol sink sequence 40, which identifies the device consuming the data. Sequence 40 begins by determining a data format in block 42. The protocol sink sequence can be implemented on the platform or the display, as shown in two examples. A potential data source (e.g., a camera) can then be identified in block 44. Next, the data is requested from the source in a specific format, already determined in block 42, as shown in block 46. Finally, the formatted data is received and acknowledged, as shown in block 48.

[0026] The metadata used for communication between the platform and image processing devices can be in various formats, including XML. The protocol metadata can implement the syntax of software metadata commands and statuses. A selected set of protocol policies can be used to enable bidirectional communication between platforms and image processing devices.

[0027] A wide variety of commands can be developed, or standard commands can be provided. Each command can return a status code indicating success, failure, or an error code, in addition to returning any other useful requested information. As a first example of a command, an image preprocessing pipeline request or response can specify elements such as sharpening, contrast enhancement, or HISTEQ. An image format request or response command can specify the color space to be used, such as RGB; the patterns that can be used, such as Bayer, YUV 422, YUV 444, HSD, Luv, or similar; and dimensions, such as whether x / y dimensions should be used.

[0028] Another possible command, an optimization command, involves a request and a response to specify the devices and applications for which a camera optimizes both the image and metadata. This can be based on a published list of device model profiles for a list of known devices participating in this technology. These profiles can be stored in the camera or retrieved by the camera from a network or storage device on demand. This arrangement allows the camera to further optimize the image with respect to a device's color gamut or for a given application such as facial recognition.Therefore, standard device configurations can be registered and stored online in a well-known location, which makes it possible to use a standardized protocol of the type described here to obtain device configurations or to set device configurations used by this protocol.

[0029] Another potential command is an interest point request or response to specify the types of interest points desired. Yet another example of a command is a describer request and response to specify the type of region describers around the interest points. Other commands include light source requests and responses to specify the light source colors and direction vectors to return. Other examples include device color model requests and responses to return the device's color model. The request or response can include a combination of desired information, such as a mathematical model of the device's color gamut and low-level virtual machine (LLVM) code like Java bytecode. A further command is a depth map request and response to specify the format to be returned in the depth map.Possibilities exist in protocol execution forms to specify computer storage data formats including integer and floating-point precision, integer 8 or integer 16, x / y dimensions of image areas and property formats of depth maps to include polygon 3D mesh points and point or pixel depth maps.

[0030] The following diagram provides example commands with descriptions and an Extensible Markup Language (XML) sample execution form. Each command can return a status code indicating success, failure, or an error code, in addition to returning any other useful requested information. command Description XML sample execution form Image preprocessing pipeline Elements such as sharpening, contrast enhancement, HISTEQ <processing> <pipeline>< / pipeline> < / processing> RESPONSE TO INQUIRY etc. specify. <sharpen_type1 / > <histeq / > Image format REQUEST RESPONSE RBG, BAYER, YUV422, YUV444, HSV, Luv etc. specify dimension (x / y dimension) <imageformat><RGB 16bbp / ><dimension = „640x480" / >< / imageformat> Optimize REQUEST RESPONSE The devices and apps specify for which a camera optimizes both 1) the images and 2) metadata. This is based on a published list of profiles, device models, etc., for a list of known devices participating in the VDP standard, whose profiles are integrated into the camera or can be retrieved by the camera from a network or storage device on demand. This allows the camera to optimize the image for a device's color gamut or for a given application such as facial recognition. <optimizeimage><HP_printer_modell23 / ><Sharp_3d_display_model_123 / ><Face_recognition_app_from_Metao / >< / optimizeimage> Points of Interest Inquiry Response Specify the type of points of interest desired (harrisCorner, Canny, LBP, etc.). <interestpoints> <canny / > < / interestpoints> Describer / Inquiry Response Specify the type of region describer around the points of interest (ORB, HOG, SIFT, GLOH, DAISY, etc.) <descriptor> <sift / > <gloh / > < / descriptor> LightSources INQUIRY RESPONSE Specify that light colors and directed vectors should be returned. <lightsources> <request3dlightvector / > <requestlightcolor / > < / lightsources> Device color model INQUIRY RESPONSE Returning the device's color model is a combination of desired information such as the mathematical model in LLVM of Java bytes code of the color range. <devicecolormodel> <mathematicalmodel / > <colorgamut / > < / DEVICEcOLORmODEL< / devicecolormodel> Device values ​​[with white point, black point, neutral grey axis, RGB maximum values, Jch max.]. Depth map REQUEST RESPONSE Specify the format for the returned depth map (precision [int8,int16[, x / y dimension,include_polygon_3Dmesh_points, include_pixel_depth_map) <depthmap><precision = „int16" / ><include_pixel_depth_map / ><include_polygon_3dmesh / >< / depthmap> List basic elements. REQUEST RESPONSE List the basic functions that the device can perform. This can also return a compatibility level such as LEVEL 1, which means that the device supports all basic elements and capabilities at LEVEL 1. <listprimitives> <all / > <compatabilitylevelonly / > < / listprimitives> Accept basic elements REQUEST Sending Java bytecode or LLVM code to a smart device, where the code is an algorithm to be executed, assumes that the device has a processor and can accept and execute code. The code can then later be executed on command using its name. <acceptprimitive> <code =„10102399903ABBCBD123230DC"<name = „nameOfThisCode" / ><whenToRun = „at_startup" / >< / code> Execute basic element INQUIRY< / acceptprimitive> Execute a basic element by name <name = „someCoolPrimitive" / > <runprimitive> Pipeline generation REQUEST< / runprimitive> Create a pipeline from a sequence of basic elements Search methodREQUEST(splittransactionw / SearchResults) <pipeline><name = „primitive1" / >...<name = „primitive n" / > <!-- / pipeline-->< / pipeline> Specify what the smart camera should search for, what metadata it should return, how much metadata and how often the metadata should be returned. ​ <searchmethod><name = „faces_method1" / ><interestPoints = „HarrisCorner" / ><pointCount = „1000_max" / ><descriptors = „ORB" / ><descriptorCount = "100_max> / <result =„return_every_10_frames" / ><frames =„every_10 th _image_frame" / >< / searchmethod> Search targetINQUIRY(splittransactionw / SearchResults) Specify what the smart camera should search for. This allows you to tell the camera what types of vertices to search for, what types of faces to search for, what types of vehicles, etc. <searchmethod><target = „faces_method1" / < / searchmethod> Search results ANSWER(splittransactionw / SearchMethod) The results are received from the corresponding search command. The results arrive periodically as specified in the search method command. <searchresults> <descriptor id="1><Interestpoint x= „223" y=„533" / ><Descriptor vector= „100Ba..." / / >< / descriptor><imageFrame><data " ..." > < data>< / descriptor> < / searchresults>

[0031] Some versions also allow for special image processing and the creation of video analytics protocols, such as a sequential combination like sharpening the image, correcting the image color, searching for faces, sending face coordinates to the printer, or sending the image to the printer.

[0032] Another list of primitive commands specifies the basic functions the device can perform, except that the primitives request Java bytecode or LLVM code to be sent to a smart device. The code can be an algorithm to be executed and assumes the device has a processor and can accept and execute the code, which can then be executed later on command by name. The `execute primitive` command can be a request to execute a primitive by name. The `generate` command can generate a pipeline of a sequence of primitives. The `searchmethod` command can be a request to specify what the smart camera should search for, what metadata should be returned, and how much and how often the metadata should be returned. The `searchtarget` command is a request that specifies what the smart camera should search for.This allows the camera to be told, for example, what types of vertices to search for, what types of faces to look for, and what types of vehicles to search for. Ultimately, a "search results" command can be a response to receive the results from the corresponding search command. The results can arrive periodically, as specified by the search method command.

[0033] References in this description to “an embodiment” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation within the present invention. Thus, uses of the expression “an embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be introduced in other suitable forms that differ from the particular illustrated embodiment, and all such forms may be included within the claims of the present application.

[0034] Although the present invention has been described with regard to a limited number of embodiments, those skilled in the art are aware that many further modifications and variants thereof are possible. The appended claims are intended to cover all such modifications and variants that correspond to the purpose and scope of the present invention.

Claims

[1] Procedure, encompassing: Sending a message from a platform to an imaging device to specify the following: a format of image data and which image data should be transferred from the imaging device to the platform, wherein the image data belongs to an image that is to be displayed on the platform, wherein the message further contains which displayed objects in the image data should be searched for, which metadata must be transferred back and how often the metadata must be transferred back; and Receiving the image data by the platform; and Processing the received image data based on the message. [2] Method according to claim 1, including the transmission of commands from a platform to a camera to specify the processing that must take place in the camera before image data is transmitted to a platform. [3] Method according to claim 1, including sending a message from a platform to a display device to ensure that data provided to the display device is brought into a specific format. [4] Method according to claim 1, including defining a bidirectional protocol for exchanging information about image data between a platform and an imaging device. [5] Method according to claim 1, including the use of predetermined commands to specify properties of information exchanged between a platform and an imaging device for processing image data. [6] Method according to claim 1, including offloading the processing of image data from a processor to a peripheral device by providing instructions to the peripheral device to perform the processing on the peripheral device prior to transferring image data to the processor. [7] One or more non-volatile, computer-readable media that store commands which cause a computer to: To send a message from a platform to an imaging device to specify the following: a format of image data and which image data should be transferred from the imaging device to the platform, wherein the image data belongs to an image that is to be displayed on the platform, wherein the message further contains which displayed objects in the image data should be searched for, which metadata must be transferred back and how often the metadata must be transferred back; and to process the image data based on the message. [8] Media according to claim 7, further comprising storing commands to transmit commands from a platform to a camera to specify the processing that must take place in the camera before image data is transmitted to a platform. [9] Method according to claim 7, further comprising storing commands to exchange messages between a platform and a display device to ensure that data provided to the display device is brought into a specific format. [10] Media according to claim 7, further comprising storing commands to establish a bidirectional protocol for exchanging information about image data between a platform and an imaging device. [11] Media according to claim 7, further comprising storing commands to use predetermined commands to specify properties of information exchanged between a platform and image data for processing image data. [12] Media according to claim 7, further comprising storing instructions to offload the processing of image data from a processor to a peripheral device by providing instructions to the peripheral device to perform the processing at the peripheral device prior to transferring image data to the processor. [13] Device comprising: a processor, where the processor is configured to: for sending a message to a platform, wherein the message specifies: a format of image data; and which image data are to be transmitted from an image device to the platform, wherein the image data belong to an image to be displayed on the platform, wherein the message further specifies which displayed objects are to be searched for in the image data, which metadata must be transmitted back, and how often the metadata must be transmitted back; and the processor is further configured to for generating image data for the platform in a format specified by the message; and the device further includes a memory that is coupled to this processor, the memory being designed to store the image data. [14] Device according to claim 13, wherein the device is a camera. [15] Device according to claim 13, wherein the device is a printer. [16] Device according to claim 13, wherein the device is a display. [17] Device according to claim 13, wherein the processor uses a bidirectional protocol for exchanging information about image data. [18] Device according to claim 13, wherein the processor uses predetermined instructions to specify properties of information exchanged for processing image data. [19] Device comprising: a processor to send a message to an imaging device specifying an image data format and which image data is to be transferred from the imaging device to the platform, wherein the image data belongs to an image to be displayed on the platform, wherein the message further specifies which displayed objects are to be searched for in the image data, which metadata must be transferred back, how often the metadata must be transferred back; and to receive image data in the format specified in the message; and a memory that is coupled with this processor. [20] Device according to claim 19, wherein the device is a mobile phone. [21] Device according to claim 19, wherein the processor sends a message to a camera specifying an image data format for data from the camera. [22] Device according to claim 19, wherein the processor sends the message via a bidirectional protocol for exchanging image data. [23] Device according to claim 22, wherein the processor uses predetermined instructions with the imaging device. [24] Device according to claim 15, wherein the processor unloads an image processing task using the message to an image device. [25] Device according to claim 19, wherein the processor provides data to a display in a format specified in a message received from the display.

Images