SYSTEM AND METHOD FOR CREATING AND RUNNING A COMPUTER VISION PIPELINE FOR IMAGE AND / OR VIDEO PROCESSING - Patent application

Abstract

A computer vision pipeline is created and executed to process images and / or videos. A user may select blocks for a computer vision pipeline, including an image transformation block to transform an image and / or video, and a prediction block to detect objects or classes of objects in the image and / or video. The user may specify an input source for the image and / or video and an output destination for the computer vision pipeline. When activated, the computer vision pipeline receives input images and / or videos. The input images and / or videos are transformed using an image transformation block to generate transformed images and / or videos. Objects or classes of objects in the transformed images and / or videos are detected using a prediction block to generate detected objects or classes of objects. The detected objects or classes of objects are sent to the output destination.

Description

[Technical field]

[0001] One or more embodiments of the present invention relate generally to the generation and execution of a computer vision pipeline for processing images and / or videos, and more particularly to the generation of a computer vision pipeline for processing images and / or videos that utilizes an image transformation block that transforms the image and / or video, and a prediction block that detects objects or classes of objects within the image and / or video. [Background technology]

[0002] Computer vision generally refers to the use of computing devices to analyze images and / or videos to gain a high level of understanding of the images and / or videos. Applications of computer vision include detecting product defects on assembly lines, monitoring personal protective equipment (PPE) compliance on construction sites, and monitoring and detecting gas and fluid leaks.

[0003] Computer vision models can be used to process images and / or videos. Software engineers and data scientists can implement computer vision models using software such as Jupyter Notebook from Project Jupyter. Summary of the Invention [Means for solving the problem]

[0004] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium that includes executable instructions, The executable instructions are executable by one or more processors to perform a method, The above method: receiving a request to create a computer vision pipeline for processing images and / or videos; receiving an image and / or video input source for said computer vision pipeline; displaying a plurality of blocks for selection, said plurality of blocks comprising: a plurality of image transformation blocks, each of which transforms an image and / or video to generate a transformed image and / or video; a plurality of prediction blocks, one of which detects an object or a class of objects in an image and / or video to generate a detected object or class of objects; including, steps; receiving a selection of one of said image transformation blocks and configuration information of said image transformation block; adding said image transformation block to said computer vision pipeline; receiving a selection of one of said prediction blocks and configuration information of said prediction block; adding the prediction block to the computer vision pipeline; receiving an output of the computer vision pipeline; receiving a request to activate the computer vision pipeline; activating said computer vision pipeline; receiving an input image and / or video from said input source; transforming the input image and / or video using the image transformation block to generate a transformed image and / or video; detecting an object or class of objects in the transformed image and / or video using the prediction blocks to generate a detected object or class of objects; and sending said detected object or class of object to said output destination. Includes.

[0005] In some aspects, the techniques described herein include: The method further comprises the step of displaying a user interface element onto which one of the blocks can be dragged and dropped; the step of receiving the selection of the image conversion block and configuration information of the image conversion block includes the step of receiving a drag and drop of the image conversion block onto the user interface element; wherein the step of receiving a selection of the predictive block and configuration information for the predictive block includes receiving a drag and drop of the predictive block onto the user interface element.

[0006] In some aspects, the techniques described herein include: The method further includes receiving a computer vision pipeline type, the computer vision pipeline type being one of a batch type or a streaming type; If the computer vision pipeline type is the batch type, the method further comprises receiving a processing schedule, and the receiving of input images and / or video from the input source comprises receiving input images and / or video from the input source according to the processing schedule; When the computer vision pipeline type is the streaming type, receiving input images and / or video from the input source includes continuously receiving input images and / or video from the input source.

[0007] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium, wherein the multiple image transformation blocks include an image crop block that crops images and / or videos, and an image resize block that resizes images and / or videos.

[0008] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium, wherein the plurality of prediction blocks includes a tiled model block that divides an image and / or video into a plurality of images for object or object class detection, detects an object or class of objects in the plurality of images, and recombines the plurality of images to generate a transformed image and / or video, and generates the transformed image and / or video and the object or object class detection.

[0009] In some aspects, the techniques described herein may further comprise a step of: a label detection block for detecting objects in an image and / or video and generating one or more label descriptions for the detected objects; a landmark detection block that detects landmarks in the image and / or video and generates a description of the detected landmarks; A logo detection block that detects logos in images and / or videos and returns a description of the detected logos; The present invention relates to a non-transitory computer-readable medium, including:

[0010] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium, wherein the method further includes displaying an overview of the computer vision pipeline, the overview including the input sources, the image transform block, the prediction block, an order of the image transform block and the prediction block within the computer vision pipeline, and the output destination.

[0011] In some aspects, the techniques described herein relate to a system that includes at least one processor and a memory containing instructions, The instructions, by the at least one processor: receiving a request to create a computer vision pipeline for processing images and / or videos; receiving an image and / or video input source for the computer vision pipeline; Displaying a plurality of blocks for selection, wherein the plurality of blocks include: a plurality of image transformation blocks, each of which transforms an image and / or video to generate a transformed image and / or video; a plurality of prediction blocks, one of which detects an object or a class of objects in an image and / or video to generate a detected object or class of objects; Includes; receiving a selection of one of the image transformation blocks and configuration information of the image transformation block; Add the image transformation block to the computer vision pipeline; receiving a selection of one of the prediction blocks and configuration information of the prediction block; adding said prediction block to said computer vision pipeline; Receive the output of the above computer vision pipeline; Receive a request to activate the computer vision pipeline; Activate the computer vision pipeline above; receiving input images and / or video from said input source; transforming the input image and / or video using the image transformation block to generate a transformed image and / or video; detecting an object or class of objects in the transformed image and / or video using the prediction block to generate a detected object or class of objects; and Sending said detected object or class of object to said output destination. Therefore, it is feasible.

[0012] In some aspects, the techniques described herein include: the instructions are further executable by the at least one processor to display a user interface element onto which one of the blocks can be dragged and dropped; the instructions executable by the at least one processor to receive a selection of the image transformation block and configuration information for the image transformation block include instructions executable by the at least one processor to receive a drag and drop of the image transformation block onto the user interface element; The system relates to a method for receiving a prediction block from a plurality of inputs, the method comprising: ...

[0013] In some aspects, the techniques described herein include: The instructions are further executable by the at least one processor to receive a computer vision pipeline type, where the computer vision pipeline type is one of a batch type or a streaming type; If the computer vision pipeline type is the batch type, the instructions are further executable by the at least one processor to receive a processing schedule, and the instructions executable by the at least one processor to receive input images and / or video from the input source include instructions executable by the at least one processor to receive input images and / or video from the input source in accordance with the processing schedule; When the computer vision pipeline type is the streaming type, the instructions executable by the at least one processor to receive input images and / or video from the input source include instructions executable by the at least one processor to continuously receive input images and / or video from the input source.

[0014] In some aspects, the techniques described herein relate to a system in which the multiple image transformation blocks include an image crop block that crops images and / or videos and an image resize block that resizes images and / or videos.

[0015] In some aspects, the techniques described herein relate to a system in which the prediction blocks include a tiled model block that divides an image and / or video into multiple images for object or object class detection, detects objects or classes of objects in the multiple images, and recombines the multiple images to generate a transformed image and / or video to generate the transformed image and / or video and the object or object class detection.

[0016] In some aspects, the techniques described herein relate to a system in which the prediction blocks include a tiled model block that divides an image and / or video into multiple images for object or object class detection, detects objects or classes of objects in the multiple images, and recombines the multiple images to generate a transformed image and / or video to generate the transformed image and / or video and the object or object class detection.

[0017] In some aspects, the techniques described herein relate to a system in which the instructions are further executable by the at least one processor to display an overview of the computer vision pipeline, the overview including the input sources, the image transform block, the prediction block, the ordering of the image transform block and the prediction block within the computer vision pipeline, and the output destination.

[0018] In some aspects, the techniques described herein include: receiving a request to create a computer vision pipeline for processing images and / or videos; receiving an image and / or video input source for said computer vision pipeline; displaying a plurality of blocks for selection, said plurality of blocks comprising: a plurality of image transformation blocks, each of which transforms an image and / or video to generate a transformed image and / or video; a plurality of prediction blocks, one of which detects an object or a class of objects in an image and / or video to generate a detected object or class of objects; including, steps; receiving a selection of one of said image transformation blocks and configuration information of said image transformation block; adding said image transformation block to said computer vision pipeline; receiving a selection of one of said prediction blocks and configuration information of said prediction block; adding the prediction block to the computer vision pipeline; receiving an output of the computer vision pipeline; receiving a request to activate the computer vision pipeline; activating said computer vision pipeline; receiving an input image and / or video from said input source; transforming the input image and / or video using the image transformation block to generate a transformed image and / or video; detecting an object or class of objects in the transformed image and / or video using the prediction blocks to generate a detected object or class of objects; and sending said detected object or class of object to said output destination. The present invention relates to a method comprising the steps of:

[0019] In some aspects, the techniques described herein include: The method further includes the step of displaying a user interface element onto which one of the blocks can be dragged and dropped; the step of receiving the selection of the image conversion block and configuration information of the image conversion block includes the step of receiving a drag and drop of the image conversion block onto the user interface element; wherein the step of receiving a selection of the predictive block and configuration information for the predictive block includes receiving a drag and drop of the predictive block onto the user interface element.

[0020] In some aspects, the techniques described herein include: receiving a computer vision pipeline type, the computer vision pipeline type being one of a batch type or a streaming type; If the computer vision pipeline type is the batch type, further comprising receiving a processing schedule, wherein the receiving of input images and / or video from the input source comprises receiving input images and / or video from the input source according to the processing schedule; When the computer vision pipeline type is the streaming type, receiving the input images and / or video from the input source comprises continuously receiving the input images and / or video from the input source.

[0021] In some aspects, the techniques described herein relate to a method in which the image transformation blocks include an image crop block that crops images and / or videos and an image resize block that resizes images and / or videos.

[0022] In some aspects, techniques described herein relate to methods where the prediction blocks include a tiled model block that divides an image and / or video into multiple images for object or object class detection, detects objects or classes of objects in the multiple images, and recombines the multiple images to generate a transformed image and / or video to generate the transformed image and / or video and the object or object class detection.

[0023] In some aspects, the techniques described herein may further comprise a step of: a label detection block for detecting objects in an image and / or video and generating one or more label descriptions for the detected objects; a landmark detection block that detects landmarks in the image and / or video and generates a description of the detected landmarks; A logo detection block that detects logos in images and / or videos and returns a description of the detected logos; The present invention relates to a method comprising the steps of: [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an example environment in which a computer vision pipeline can be created and executed in some embodiments. [Diagram 2] FIG. 2 illustrates components of a computer vision pipeline system in some embodiments. [Figure 3-1] FIG. 3 illustrates a method for creating and running a computer vision pipeline in some embodiments. [Figure 3-2] FIG. 3 illustrates a method for creating and running a computer vision pipeline in some embodiments. [Figure 4] FIG. 4 shows an example interface for receiving a request to create a new computer vision pipeline in some embodiments. [Figure 5A] FIG. 5A shows an example interface for receiving a computer vision pipeline name, a computer vision pipeline type, and a computer vision pipeline input type in some embodiments. [Figure 5B] FIG. 5B shows another example interface for receiving a computer vision pipeline name, a computer vision pipeline type, and a computer vision pipeline input type in some embodiments. [Figure 6A] FIG. 6A illustrates an example interface for receiving image and / or video input sources for a computer vision pipeline in some embodiments. [Figure 6B]FIG. 6B illustrates another example interface for receiving image and / or video input sources for a computer vision pipeline in some embodiments. [Figure 6C] FIG. 6C illustrates an example interface for receiving a computer vision pipeline processing schedule in some embodiments. [Figure 7] FIG. 7 illustrates an example interface for receiving a selection of a block of a computer vision pipeline in some embodiments. [Figure 8A] FIG. 8A illustrates an example interface for receiving configuration information for an image crop block for cropping an image and / or video in some embodiments. [Figure 8B] FIG. 8B illustrates an example interface for receiving configuration information for an image resizing block for resizing images and / or videos in some embodiments. [Figure 9A] FIG. 9A illustrates an example interface for receiving configuration information for a single model block in some embodiments. [Figure 9B] FIG. 9B illustrates an example interface for receiving configuration information for a tiled model block in some embodiments. [Figure 9C] FIG. 9C illustrates another example interface for receiving configuration information for a tiled model block in some embodiments. [Figure 9D] FIG. 9D illustrates an example interface for receiving configuration information for a generic object detection block in some embodiments. [Figure 9E] FIG. 9E illustrates an example interface for receiving configuration information for an object tracking block in some embodiments. [Figure 9F] FIG. 9F illustrates an example interface for receiving configuration information for a label detection block in some embodiments. [Figure 9G]FIG. 9G shows an example interface for receiving landmark detection block configuration information in some embodiments. [Figure 9H] FIG. 9H illustrates an example interface for receiving configuration information for the logo detection block in some embodiments. [Figure 9I] FIG. 9I illustrates an example interface for receiving configuration information for an image properties block in some embodiments. [Figure 10] FIG. 10 illustrates the example interface of FIG. 7 with multiple blocks added to the computer vision pipeline in some embodiments. [Figure 11] FIG. 11 illustrates an example interface for receiving output destinations of a computer vision pipeline in some embodiments. [Figure 12] FIG. 12 shows an example interface for displaying an overview of a computer vision pipeline and receiving a request to activate the computer vision pipeline in some embodiments. [Figure 13] FIG. 13 shows an example interface for displaying an overview of a computer vision pipeline and receiving requests to modify the computer vision pipeline in some embodiments. [Figure 14] FIG. 14 illustrates an example interface for displaying an overview of multiple computer vision pipelines in some embodiments. [Figure 15] FIG. 15 illustrates an example of the use of a computer vision pipeline in some embodiments. [Figure 16A] 16A and 16B show examples of predicted block outputs in some embodiments. [Figure 16B] 16A and 16B show examples of predicted block outputs in some embodiments. [Figure 17]FIG. 17 illustrates a block diagram of an exemplary digital device according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Throughout the drawings, like reference numbers should be understood to refer to like parts, components and structures.

[0026] To implement a computer vision model using software such as Jupyter Notebook, software engineers and data scientists typically need to write software to load training data, train a computer vision model with the training data, perform inference using the trained computer vision model, and save the results of the inference. Such an approach has several drawbacks. First, it may require a certain level of software development experience. Second, computer vision models may require extensive configuration to achieve the desired objectives. Third, such an approach lacks scalability across multiple organizations, or even across multiple users within an organization. Furthermore, while data engineers are highly specialized (given the hand-coding and building nature of analytical systems), they rarely have the domain expertise required to ensure that they avoid false assumptions, identify specific features, and properly tune the output to meet business needs.

[0027] In some embodiments, the computer vision pipeline system described herein allows users to implement computer vision solutions without having to write software (e.g., as a "no-code" or "low-code" system). The computer vision pipeline system allows users with little or no software development experience to create, run, and modify computer vision pipelines for processing images and / or videos. In various embodiments, the computer vision pipeline system provides an interface that allows users to quickly and easily select blocks for a computer vision pipeline, configure the blocks, and activate the computer vision pipeline.

[0028] In some embodiments, the computer vision pipeline system may include a graphical user interface having interactive visual elements. An interactive visual element may represent a function or a combination of previously coded functions. In some embodiments, the interactive visual element may be represented as a "block" or any other representation. Although the term "block" is used herein, it will be understood that the visual element representing one or more functions may be of any shape (or combination of different shapes), media, animation, and / or other.

[0029] In some embodiments, the blocks of the computer vision pipeline include an image transformation block that transforms images and / or videos. The image transformation block may be used at the beginning of the computer vision pipeline to transform input images and / or videos. The blocks also include a prediction block that detects objects and / or classes of objects in the images and / or videos. The prediction block adds inference metadata to the images and / or videos in the computer vision pipeline.

[0030] In addition to the advantages discussed above, the computer vision pipeline system allows a user to quickly and easily specify input sources of images and / or videos, and output destinations for inferred data and / or processed images and / or videos. Additionally, the computer vision pipeline system allows a user to specify whether the computer vision pipeline processes images and / or videos continuously or on a scheduled basis, and, in the latter case, the processing schedule.

[0031] The computer vision pipeline system is scalable, due in part to its inherent ease of use, distributed processing, customizability, and / or other aspects. Furthermore, in a centralized system (such as that shown in FIG. 1), multiple users within an organization, and any number of different organizations, may be able to utilize the computer vision pipeline system to create platforms and / or applications for computer vision tasks.

[0032] It will be further appreciated that the "low-code" or "no-code" systems discussed herein enable a variety of users with different experience (i.e., not just specialized data engineers or those with extensive coding experience) to create computer vision pipelines via the computer vision interface, thereby enabling users with domain expertise to leverage their understanding of the business, challenges, and solutions to create computer vision applications, platforms, and / or processes that are tailored to the specific needs of the business.

[0033] Furthermore, by utilizing already coded "blocks" (i.e., functionality), different businesses and users can leverage the system and flexibility without having to code different processes individually. As a result, significant time and computer resources are saved by avoiding the debugging, testing, updating, justifying, and documenting of new code that is typically required for every new code for a critical system. Other advantages of the computer vision pipeline system described above will become apparent.

[0034] 1 illustrates an example environment 100 in which a computer vision pipeline may be created and executed in some embodiments. The example environment 100 includes a computer vision pipeline system 104, a user system 106, an input source system 108, a destination system 110, and a communication network 112. Each of the computer vision pipeline system 104, the user system 106, the input source system 108, and the destination system 110 may be a digital device or may include any number of digital devices. A digital device is any device that includes at least one processor and memory. Digital devices are discussed further, for example, with reference to FIG. 17.

[0035] The computer vision pipeline system 104 can provide an interface for creating, activating, running, and managing computer vision pipelines. The computer vision pipeline system 104 can provide such an interface to the user system 106, allowing a user of the user system 106 to, among other things, request the computer vision pipeline system 104 to create a computer vision pipeline, select blocks of the computer vision pipeline, and request the computer vision pipeline system 104 to activate and run the computer vision pipeline. The computer vision pipeline system 104 receives input images and / or videos from an input source system 108 and processes the input images and / or videos. The computer vision pipeline system 104 can process the input images and / or videos by transforming the input images and / or videos and / or by detecting objects and / or classes of objects in the input images and / or videos. The computer vision pipeline system 104 sends the detected objects and / or classes of objects, and optionally the transformed images and / or videos, to the destination system 110.

[0036] The user system 106 can display interfaces to a user that allow the user to request the creation of a computer vision pipeline, select blocks in the computer vision pipeline, and request activation and execution of the computer vision pipeline. The user system 106 can also display interfaces to a user that allow the user to specify input sources and output destinations for the computer vision pipeline, configure blocks in the computer vision pipeline, and manage the computer vision pipeline.

[0037] The input source system 108 may be or include any system capable of providing images and / or video. The input source system 108 may be or include cloud storage providers such as Google Cloud Storage and Amazon S3, messaging services such as Google Pub / Sub, and local data storage. The input source system 108 may be or include a camera or sensor (e.g., a security camera) that continuously streams images and / or video. The input source system 108 may be or include a system that provides satellite and / or video images and / or videos and aerial images and / or videos. The input source system 108 transmits images and / or videos to the computer vision pipeline system 104.

[0038] The destination system 110 may be or may include cloud storage providers such as Google Cloud Storage and Amazon S3, messaging services such as Google Pub / Sub, and local data storage. The destination system 110 may receive detected objects and / or classes of objects, and optionally processed images and / or videos, from the computer vision pipeline system 104.

[0039] In some embodiments, the communication network 112 may represent one or more computer networks (e.g., a LAN, a WAN, and / or the like). The communication network 112 may provide communication between any of the computer vision pipeline system 104, the user systems 106, the input source systems 108, and the output destination systems 110. In some implementations, the communication network 112 includes computer devices, routers, cables, applications, and / or other network topologies. In some embodiments, the communication network 112 may be wired and / or wireless. In various embodiments, the communication network 112 may include the Internet and one or more networks that may be public networks, private networks, IP-based networks, non-IP-based networks, etc.

[0040] 1 illustrates the environment 100 as including only one each of the computer vision pipeline system 104, the user system 106, the input source system 108, the destination system 110, and the communication network 112. However, the environment 100 may include more than one of any of the computer vision pipeline system 104, the user system 106, the input source system 108, the destination system 110, and the communication network 112.

[0041] 2 illustrates components of the computer vision pipeline system 104 in some embodiments. The computer vision pipeline system 104 includes a communications module 202, a user interface module 204, a display module 206, a computer vision pipeline creation module 208, a computer vision pipeline activation module 210, an image transformation module 212, an object / class detection module 214, a computer vision pipeline execution module 216, and data storage 220.

[0042] The communications module 202 can send and receive requests and / or data between the computer vision pipeline system 104 and any of the user systems 106, the input source systems 108, and the destination systems 110. The communications module 202 can receive requests and / or data from the user systems 106, the input source systems 108, and the destination systems 110. The communications module 202 can also send requests and / or data to the user systems 106, the input source systems 108, and the destination systems 110.

[0043] The user interface module 204 can receive requests to create, modify, and run a computer vision pipeline. The display module 206 can display and / or provide a display interface for a user to interact with to create, activate, run, and modify a computer vision pipeline. The computer vision pipeline creation module 208 can add user-selected blocks to the computer vision pipeline.

[0044] The computer vision pipeline activation module 210 can activate the computer vision pipeline. The image transformation module 212 can have the image transformation block transform the image and / or video to generate a transformed image, if the computer vision pipeline includes an image transformation block. The object / class detection module 214 can have the prediction block detect an object or class of object in the image and / or video, if the computer vision pipeline includes a prediction block, to generate a detected object or class of object.

[0045] The computer vision pipeline execution module 216 can execute the computer vision pipeline. The computer vision pipeline execution module 216 can convert the computer vision pipeline into a directed acyclic graph (DAG), such as an Apache Beam-based DAG, and execute the computer vision pipeline.

[0046] Data storage 220 may include data stored, accessed, and / or modified by any of the modules of the computer vision pipeline system 104, such as user-created computer vision pipeline, image and / or video, and inference data. Data storage 220 may include any number of data storage structures, such as tables, databases, and / or lists, and / or other data storage structures.

[0047] A module may be hardware, software, firmware, or any combination thereof. For example, each module may include functions performed by dedicated hardware (e.g., an Application-Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), etc.), software, instructions stored in ROM, and / or any combination thereof. Software may be executed by one or more processors. Although only a limited number of modules are shown in FIG. 2, there may be any number of modules. Furthermore, an individual module may perform any number of functions, including the functions of multiple modules shown herein.

[0048] FIG. 3 illustrates a method 300 for creating and running a computer vision pipeline, according to some embodiments. The computer vision pipeline system 104 can perform the method 300. The method 300 begins at step 302, where the user interface module 204 receives a request to create a computer vision pipeline for processing images and / or video. FIG. 4 illustrates an example interface 400 for receiving a request to create a new computer vision pipeline, according to some embodiments. The display module 206 can display the interface 400, as well as other interfaces described herein, to a user of the user system 106. The interface 400 includes a navigation element 402 that indicates to the user that the user is creating and / or managing a computer vision pipeline. The interface 400 also includes a button 404 labeled "Create New Pipeline." The user can select the button 404 to request that the computer vision pipeline system 104 create a computer vision pipeline. The user system 106 sends the request to the computer vision pipeline system 104.

[0049] Returning to FIG. 3 , in step 304, the user interface module 204 receives a computer vision pipeline type. The computer vision pipeline type may be batch or streaming. A batch type computer vision pipeline receives input images and / or video from the input source system 108 according to a processing schedule or on an on-demand basis. A streaming computer vision pipeline type receives input images and / or video continuously from the input source system 108. FIG. 5A illustrates an example interface 500 for receiving a computer vision pipeline name, a computer vision pipeline type, and a computer vision pipeline input type in some embodiments. The interface 500 includes a number of text labels that indicate to a user where the user is in the process of creating a computer vision pipeline. The number of text labels include a name / type text label 502, an input text label 504, a pipeline block text label 506, an output text label 508, and a summary text label 510. A user can select a text label to navigate between different interfaces displayed by the display module 206.

[0050] The interface 500 includes a computer vision pipeline name section 512 where a user can specify a name for the computer vision pipeline, a computer vision pipeline type section 514 where a user can specify a computer vision pipeline type, and a computer vision pipeline input type section 516 where a user can specify a computer vision pipeline input type. The computer vision pipeline type section 514 allows a user to select a patch type for the computer vision pipeline or a streaming type for the computer vision pipeline. If a user selects a batch type for the computer vision pipeline type, the computer vision pipeline system 104 will receive and process input images and / or videos from the input source system 108 according to a processing schedule or on an on-demand basis. If a user selects a streaming type for the computer vision pipeline type in the computer vision pipeline type section 514, the computer vision pipeline system 104 will continuously receive and process input images and / or videos from the input source system 108. If a user selects an image input for the computer vision pipeline input type in the computer vision pipeline input type section 516, the computer vision pipeline system 104 will receive and process images from the input source system 108. If the user selects video input for the computer vision pipeline input type in the computer vision pipeline input type section 516, the computer vision pipeline system 104 will receive and process video from the input source system 108. In some embodiments, the computer vision pipeline input type section 516 includes a combined image and video input option. In such embodiments, the computer vision pipeline system 104 can receive and process both images and / or video from the input source system 108.The user can select a button 518 labeled “Continue to Inputs” or the input text label 504 to proceed to an interface that allows the user to specify an input source system 108 .

[0051] 5B illustrates another example interface 550 for receiving a computer vision pipeline name, a computer vision pipeline type, and a computer vision pipeline input type in some embodiments. If the user selects video input for the computer vision pipeline input type in the computer vision pipeline input type section 516, the interface 550 provides a frame rate selector 552. The user can specify a frame rate in frames per second (fps) that the computer vision pipeline system 104 will use to split the video into individual frames (or images) for processing in the computer vision pipeline. In some embodiments, a frame rate of 8 fps may be used for tracking objects moving at moderate or lower speeds, such as the average speed of a human walking or running.

[0052] Returning to FIG. 3, in step 306, the user interface module 204 receives an input source of images and / or videos for the computer vision pipeline, and optionally a processing schedule if the computer vision pipeline type is a batch type. FIG. 6A illustrates an example interface 600 for receiving an input source of images and / or videos for the computer vision pipeline in some embodiments. The interface 600 includes a computer vision pipeline input source section 602, a computer vision pipeline input source name section 604, and a computer vision pipeline input source configuration section 606. A user can specify an input source in the computer vision pipeline input source section 602, which may be or may include a cloud storage provider, such as Google Cloud Storage and Amazon S3. The input source may be or may include local data storage, such as videos and / or images stored on the user system 106, or a file, such as a comma-separated-values ​​(CSV) file that contains pointers to the stored videos and / or images. A user can specify an input source name in a computer vision pipeline input source name section 604 and provide configuration information for the input source in a computer vision pipeline input source configuration section 606. Such configuration information can include the name of a particular bucket in a cloud storage provider, a folder name, and access information if the particular bucket is private. Such configuration information can also include a filter to include only images and / or videos with a particular extension or whose filename matches a wildcard search, and an input limit to specify the maximum number of images and / or videos that can be processed. A user can save the information specified for an input source by selecting a button 608 labeled "Save Input."

[0053] 6B illustrates another example interface 620 for receiving image and / or video input sources for a computer vision pipeline in some embodiments. The interface 620 includes a computer vision pipeline input source section 602, a computer vision pipeline input source name section 604, and a computer vision pipeline input source configuration section 606. A user can specify an input source in the computer vision pipeline input source section 602, which may be or may include a messaging service, such as Google Pub / Sub, or an application programming interface (API) endpoint. A user can specify an input source name in the computer vision pipeline input source name section 604 and provide configuration information for the input source in the computer vision pipeline input source configuration section 606. Such configuration information can include a messaging service project ID, a messaging service subscription, an account key, and / or other information for configuring the messaging service or API endpoint.

[0054] FIG. 6C illustrates an example interface 640 for receiving a computer vision pipeline processing schedule in some embodiments. The display module 206 displays the interface 640 when the computer vision pipeline type is a batch type. The interface 640 includes a connected inputs section 642, where one or more input sources are listed. A user can add an input source, search for an input source, edit an input source, and remove an input source in the connected inputs section 642. The interface 640 also includes a processing schedule section 644, where a user can set a processing schedule for the computer vision pipeline. The processing schedule can be upon activation of the computer vision pipeline, or hourly, daily, or weekly. A user can also select a start date and time for the computer vision pipeline. After specifying the processing schedule, a user can select a block of the computer vision pipeline by selecting a button 646 labeled "Continue to Pipeline Blocks."

[0055] Returning to FIG. 3, in step 308, the display module 206 displays a user interface element onto which blocks of the computer vision pipeline can be dragged and dropped. In step 310, the display module 206 displays a number of blocks for selection by the user. The number of blocks includes a number of image transformation blocks and a number of prediction blocks. The blocks may follow a rule that they receive images and / or video and data (e.g., JavaScript Object Notation (JSON) files) and generate images and / or video and data (e.g., JSON files). This rule allows the user to place the blocks anywhere in the computer vision pipeline.

[0056] 7 illustrates an example interface 700 for receiving a selection of a block of a computer vision pipeline in some embodiments. The interface 700 includes a user interface element 702 and a block library section 703. The block library section 703 includes a number of blocks including a number of image transform blocks. The image transform blocks transform an image and / or video to generate a transformed image and / or video. The image transform blocks include an image crop block 704 that crops the image and / or video and an image resize block 706 that resizes the image and / or video.

[0057] Returning to FIG. 3, in step 312, the user interface module 204 receives a selection of one or more blocks and configuration information for the selected one or more blocks. A user may select a block by dragging and dropping the block from the block library section 703 onto the user interface element 702. A user may select a block by selecting the "+" icon in the lower right corner of the block. FIG. 8A shows an example interface 800 for receiving configuration information for the image crop block 704. The display module 206 displays the interface 800 when a user selects the image crop block 704. The interface 800 allows a user to specify whether the image crop block 704 should center crop the image and / or video or should fixedly crop the image and / or video. A radio button 802 allows a user to specify whether the image crop block 704 should center crop or fixedly crop the image and / or video. If a user selects center crop, the user may specify the height and width of the cropped image and / or video. The user can also specify a minimum visibility for detection after cropping in section 804. Image 806 includes a crop box 808 that indicates the portion of the image that will remain after the image crop block performs a center crop. If the user selects a fixed crop, the user can specify the height and width of the cropped image and / or video and the location where the image crop block 704 will perform a fixed crop of the image and / or video. Image 812 includes a crop box 814 that indicates the portion of the image that will remain after the image crop block performs a fixed crop. The user can add the image crop block 704 with the specified configuration information to the computer vision pipeline by selecting a button 816 labeled "Add to Pipeline."

[0058] One advantage of cropping an image using the image crop block 704 is that certain images may be too large to fit into the memory of a graphics processing unit (GPU). Typically, such images may be resized to fit into the GPU memory. However, resizing an image may result in a loss of image resolution, which may make it more difficult to detect an object or class of object within the image. The image crop block 704 may center crop or hard crop the image and / or video, which may enable the cropped image and / or video to fit into the GPU memory.

[0059] FIG. 8B illustrates an example interface 850 for receiving configuration information for the image resize block 706. Images and / or videos can be resized to reduce size without losing spatial relationship information. The image resize block 706 may use algorithmic approaches such as nearest neighbor interpolation or Lanczos resampling. The display module 206 displays the interface 850 when a user selects the image resize block 706. The interface 850 allows a user to specify, using radio buttons 852, whether the image resize block 706 should perform fixed resizing of the image and / or video, which scales the image to a set dimension, or longest edge resizing, which scales the longest edge to a set dimension while maintaining the starting aspect ratio of the image and / or video. In the case of fixed resizing, the user can specify the height and width of the resized image via input 854. In the case of longest edge resizing, the user can specify the size of the longest edge via input 856. The user can select button 816 to add the image resize block 706 with the specified configuration information to the computer vision pipeline.

[0060] In addition to the image transformation blocks, the block library section 703 (FIG. 7) includes prediction blocks. The prediction blocks detect objects or classes of objects in the image and / or video and generate detected objects or classes of objects. The prediction blocks include a single model block 708, a tiled model block 710, a general object detection block 712, an object tracking block 714, an optical character recognition block 715, a label detection block 716, a landmark detection block 718, a logo detection block 720, and an image properties block 722. FIG. 9A illustrates an example interface 900 for receiving configuration information for the single model block 708 in some embodiments. The interface 900 includes a selector 901 that allows a user to select a machine learning model and a selector 902 for selecting a version of the machine learning model. The interface 900 also includes a selector 903 that allows a user to select a class of object that the single model block 708 detects in the image and / or video. The interface 900 also includes a confidence threshold slider 904 that allows the user to specify a minimum confidence to include as a positive detection. The image 905 includes multiple cars, and the number of bounding boxes 906 around the cars varies depending on the specified minimum confidence. The user can select button 816 to add the single model block 708 with the specified configuration information to the computer vision pipeline.

[0061] FIG. 9B illustrates an example interface 910 for receiving configuration information for the tiled models block 710 in some embodiments. The tiled models block 710 splits an image and / or video into multiple images for object or object class detection, detects objects or classes of objects in the multiple images, recombines the multiple images to generate a transformed image and / or video, and generates the transformed image and / or video and object or object class detection. A user can select the tiled models block 710 to detect small objects in high-resolution images such as satellite images. Typical models may not be able to detect extremely small objects in such images. The tiled models block 710 can sweep the image and / or video using a sliding window pattern with a fixed tile size to perform detection at a much lower resolution.

[0062] The interface 910 has the same elements as the interface 900 of FIG. 9A. The interface 910 also has a radio button 911 that allows the user to specify whether the tiled model block 710 uses predefined or custom tiling. If the user selects predefined tiling, the user can specify the tile size using the radio button 912. The image 913 includes a small tile, a medium tile, and a large tile. FIG. 9C shows another example interface 920 for receiving configuration information for the tiled model block 710 in some embodiments. In the interface 920, selecting custom tiling via the radio button 911 causes the display module 206 to display a custom tile size input 921 and a custom tile overlap input 922. In the interface 910 or the interface 920, the user can select the button 816 to add the tiled model block 710 with the specified configuration information to the computer vision pipeline.

[0063] FIG. 9D illustrates an example interface 930 for receiving configuration information for the generic object detection block 712 in some embodiments. A user can select a model using selector 901 and a class using selector 903. The generic object detection model can be a pre-trained model that can be used to detect multiple classes of generic objects, such as 80 or more classes (e.g., aircraft, person, sandwich classes). The generic object detection model can be trained on the COCO dataset, which is a dataset format that supports labeling of objects using feature points, rectangles, and polygons. A user can select button 816 to add the generic object detection block 712 with the specified configuration information to the computer vision pipeline.

[0064] FIG. 9E illustrates an example interface 940 for receiving configuration information for the object tracking block 714 in some embodiments. Object tracking includes the ability to create and track object IDs across multiple video frames. Object tracking is generally highly configurable and may need to be fine-tuned for a particular video due to differences in frame rate, object speed, performance requirements, and similarity of observed objects. Object tracking performance can be measured by how well an object is tracked by assigning the correct ID across multiple video frames. Object tracking may use polygons with intersection over union (IoU) to track certain objects such as animals. The interface 940 includes a selector 941 that allows the user to select the model of the object tracking block 714 to use, a selector 942 to select the label type as either bounding box or polygon, and a selector 943 to select the class of objects that the object tracking block 714 will detect. The user can also use radio buttons 944 to specify whether to track fast moving objects or slow moving objects. The user can select button 816 to add the object tracking block 714 with the specified configuration information to the computer vision pipeline.

[0065] FIG. 9F illustrates an example interface 950 for receiving configuration information for the label detection block 716 in some embodiments. The label detection block 716 detects objects in an image and / or video and generates one or more label descriptions for the detected objects. In some embodiments, the label detection block 716 uses APIs of third-party systems, such as the Label Detection Google Cloud Vision API, to detect objects in the image and / or video, such as products, animal species, scenes, places, actions, and activities. The label detection block 716 returns label descriptions and confidence scores for the detected objects. The interface 950 includes an input 952 that allows a user to specify a maximum number of detections for each image and / or video. The user can select a button 816 to add the label detection block 716 with the specified configuration information to the computer vision pipeline.

[0066] FIG. 9G illustrates an example interface 960 for receiving configuration information for the landmark detection block 718 in some embodiments. The landmark detection block 718 detects landmarks, such as iconic landmarks, historical landmarks, or natural structures. The landmark detection block 718 returns a description, bounding box, and location coordinates of the detected landmark entities. In some embodiments, the landmark detection block 718 detects landmarks in the images and / or videos using APIs of third-party systems, such as the Detect Landmarks Google Cloud Vision API. Although not shown in FIG. 9G, the interface 960 may include an input that allows a user to specify a maximum number of detections for each image and / or video. The user can select button 816 to add the landmark detection block 718 with the specified configuration information to the computer vision pipeline.

[0067] FIG. 9H illustrates an example interface 970 for receiving configuration information for the logo detection block 720 in some embodiments. The logo detection block 720 detects logos, such as product logos or company logos, in images and / or videos and returns a description of the detected logos. In some embodiments, the logo detection block 720 detects logos in images and / or videos using APIs of third party systems, such as the Detect Logos Google Cloud Vision API. Although not shown in FIG. 9H, the interface 970 may include an input that allows a user to specify a maximum number of detections for each image and / or video. A user can add the logo detection block 720 to the computer vision pipeline by selecting button 816.

[0068] 9I illustrates an example interface 980 for receiving configuration information for the image properties block 722 in some embodiments. The image properties block 722 detects properties and / or attributes of the image and / or video, such as color. The interface 980 includes an input 982 that allows a user to specify a maximum number of results for the image and / or video. The image properties block 722 collapses the color classification of the image and / or video into a desired number of primary colors specified in the input 982. In some embodiments, the image properties block 722 detects primary properties and / or attributes of the image and / or video using an API of a third party system, such as the Detect Image Properties Google Cloud Vision API.

[0069] Returning to Figure 7, optical character recognition block 715 can be used to perform optical character recognition (OCR) to detect characters such as letters and numbers in images and / or videos. The block library section 703 also includes a custom filter block 724. The custom filter block 724 allows a user to specify certain conditions that are executed between other blocks in the computer vision pipeline. For example, a user can specify that only images and / or videos in which automobiles are detected should proceed to the next block in the computer vision pipeline.

[0070] The computer vision pipeline system 104 may include blocks other than those described with reference to Figures 7-10, for example. Such blocks may include a two-stage block that includes two models operating in series. Another block may be a distortion block that corrects for fisheye lenses. Another block may be a person anonymization block that blurs or obscures human faces and / or entire bodies. Another block may be a scene detection block that detects in which scene a certain frame of the video is located, which may be useful for movies or other edited videos. Another block may be a motion detection block that determines whether there is motion in a particular frame of the video.

[0071] Returning to Figure 3, at step 314, the computer vision pipeline creation module 208 adds the selected blocks to the computer vision pipeline. Figure 10 illustrates the example interface 700 of Figure 7 with multiple blocks added to the computer vision pipeline in some embodiments. After the user has finished adding blocks to the computer vision pipeline, the user can select a button 726 labeled "Continue to Outputs" to proceed to an interface for selecting output destinations for the computer vision pipeline. Returning to Figure 3, at step 316, the user interface module 204 receives the output destinations.

[0072] FIG. 11 illustrates an example interface 1100 for receiving a computer vision pipeline output destination in some embodiments. The interface 1100 includes an output type section 1102 that allows a user to select an output destination, and an output name section 1104 that allows a user to specify a name for the output destination. The interface 1100 also includes an output data section 1106 that allows a user to specify whether the computer vision pipeline system 104 sends images and / or video to the output destination, and a configuration section 1108 that allows a user to specify configuration information for the output destination. Such configuration information may include a specific bucket name of a cloud storage provider, a folder name, and access information if the specific bucket is private. After providing an output destination, the user may save the output destination and proceed to an overview interface by selecting a button 1110 labeled "Save Output."

[0073] FIG. 12 illustrates an example interface 1200 for displaying an overview of a computer vision pipeline and receiving a request to activate a computer vision pipeline, according to some embodiments. The interface 1200 includes: a computer vision pipeline overview section 1202 that displays the blocks in the computer vision pipeline; and a pipeline details section 1204 that displays the computer vision pipeline name, the computer vision pipeline type, the data type, and, if the computer vision pipeline type is video, the computer vision pipeline frame rate. The interface 1200 also includes: an input sources section 1206 that displays details about the computer vision pipeline input sources; a blocks section 1208 that displays details about the blocks in the computer vision pipeline; and a destinations section 1210 that displays details about the computer vision pipeline output destinations. Each section has a text link with the label "Edit" that allows a user to edit the configuration of that particular section. The interface 1200 also includes a button 1212 labeled "Activate Pipeline." Upon user selection, the user interface module 204 receives a request to activate the computer vision pipeline (step 318 in FIG. 3 ), and the computer vision pipeline activation module 210 activates the computer vision pipeline (step 320 in FIG. 3 ). After activation, the computer vision pipeline execution module 216 can begin executing the computer vision pipeline.

[0074] In various embodiments, the computer vision pipeline execution module 216 can utilize an application programming interface (API) to execute the computer vision pipeline. The API can support multiple image and video data types, multiple pipeline types, such as batch or streaming (or edge in some embodiments), multiple input sources, such as cloud storage buckets and messaging services, and multiple output destinations, such as cloud storage and messaging services. The computer vision pipeline execution module 216 can convert the computer vision pipeline into a directed acyclic graph (DAG), such as an Apache Beam-based DAG, to execute the computer vision pipeline. The computer vision pipeline execution module 216 can create a uniform resource locator (URL) for each image and / or video frame as a signed URL and pass the signed URL to a queue. The computer vision pipeline execution module 216 can then distribute the signed URLs from the queue to multiple digital devices operating in parallel. The multiple digital devices can perform image and / or video transformation and object or object class detection in parallel and write the detection results to a JSON file. The digital devices can then send output to a destination system 110 .

[0075] Returning to FIG. 3 , in step 322, the communication module 202 receives input images and / or videos from the input source system 108. In step 324, if the computer vision pipeline includes an image transformation block, the image transformation module 212 has the image transformation block transform the images and / or videos to generate transformed images and / or videos. Also in step 324, if the computer vision pipeline includes a prediction block, the object / class detection module 214 has the prediction block detect objects or classes of objects in the images and / or videos to generate detected objects or classes of objects. In step 326, the communication module 202 sends the detected objects or classes of objects to the destination system 110. In step 328, the communication module 202 sends the transformed images and / or videos to the destination system 110.

[0076] In step 330, the display module 206 displays an overview of the computer vision pipeline. FIG. 13 illustrates an example interface 1300 for displaying an overview of the computer vision pipeline and receiving a request to modify the computer vision pipeline in some embodiments. The interface 1300 is generally similar to the interface 1200 of FIG. 12. The blocks section 1208 may include the relative costs of the blocks of the computer vision pipeline and the estimated cost of running the computer vision pipeline. The interface 1300 also includes: an indicator 1312 that indicates the status of the computer vision pipeline; a button 1314 labeled "Stop" that allows a user to request that the computer vision pipeline execution module 216 stop executing the computer vision pipeline; and a button 1316 labeled "Edit Pipeline" that allows a user to request that modifications be made to the computer vision pipeline. A user can request modifications to be made to the computer vision pipeline by selecting the button 1316 or the "Edit" text link in the interface 1300. Returning to Figure 3, in step 332, the computer vision pipeline creation module 208 receives the modifications to the computer vision pipeline. In step 334, the computer vision pipeline creation module 208 modifies the computer vision pipeline according to the requested modifications. In step 336, the display module 206 displays an overview of the computer vision pipeline and, optionally, overviews of other computer vision pipelines.

[0077] 14 illustrates an example interface 1400 for displaying an overview of multiple computer vision pipelines in some embodiments. The interface 1400 includes a list section 1402 that lists the computer vision pipelines that the user has created, their names, input types, types, status, number of inputs and outputs, creation date, and actions. The list section 1402 includes a text link 1404 labeled "Stop" that allows the user to request that the computer vision pipeline execution module 216 stop executing the computer vision pipeline. The list section 1402 also includes a text link 1406 labeled "Delete" that allows the user to request that the computer vision pipeline creation module 208 delete the computer vision pipeline. The interface 1400 also includes a button 1408 labeled "Create New Pipeline" that allows the user to request that a new computer vision pipeline be created.

[0078] An example of the use of the computer vision pipeline system and associated methods described herein is as follows: A hide, which may be a male calf hide or a cow hide, is tattooed (or similar, such as a brand) with numbers and letters, respectively. A camera may capture images and / or videos of the hide. A user may request that the computer vision pipeline system 104 create a computer vision pipeline for processing the captured images and / or videos. A user may specify an input source for the images and / or videos. A user may detect the tattoo and identify a bounding box around the tattoo in the images and / or videos by selecting the Single Model block 708 as the first block of the computer vision pipeline. A user may crop the images and / or videos to fit (or slightly larger than) the bounding box by selecting the Image Crop block 704 as the second block of the computer vision pipeline. A user may perform OCR on the cropped images and / or videos to detect letters in the cropped images and / or videos by selecting the Optical Character Recognition block 715 as the third block of the computer vision pipeline. The computer vision pipeline system 104 can then generate a JSON file containing the detected characters and send the JSON file and, optionally, the cropped image and / or video to the destination system 110. A human labeler can then manually review the detected characters in the image and / or video. A user may use the computer vision pipeline system 104 in the manner described above to generate a ground truth dataset for further training of a machine learning model and / or for other purposes. Additionally or alternatively, a user may select the single model block 708 as the first block of the computer vision pipeline and the image crop block 704 as the second block of the computer vision pipeline, which crops around the detected characters.

[0079] FIG. 15 illustrates an example of the use of a computer vision pipeline in some embodiments. The communication module 202 receives an input image of a hide from the input source system 108. In FIG. 15, the image 1500 shows a bounding box 1502 around a tattoo on the hide. The single model block 708 identifies the bounding box 1502. The image crop block 704 then crops the image 1500 to generate an image 1550. The image 1550 shows four bounding boxes: a bounding box 1504 around the character "3", a bounding box 1506 around the character "9", a bounding box 1508 around the character "X", and a bounding box 1510 around the character "9". The optical character recognition block 715 detects each character and identifies a bounding box for each character. The communication module 202 then transmits the detected characters to the destination system 110.

[0080] Another example use of the computer vision pipeline system and associated methods described herein is as follows: A live animal may need to be tracked as it moves, for example, in an enclosure, across a field, or along a track. Similarly, a dead animal may need to be tracked as it moves, for example, on a conveyor belt, hook, or other device. A camera may capture video of the live animal or dead animal. A user may request that the computer vision pipeline system 104 create a computer vision pipeline to process the captured video. A user may specify an input source for the video. A user may detect the live animal or dead animal by selecting the single model block 708 as the first block of the computer vision pipeline. A user may track the live animal or dead animal across multiple video frames by selecting the object tracking block 714 as the second block. The computer vision pipeline system 104 may then generate a JSON file that includes the tracked live animal or dead animal.

[0081] Another example use of the computer vision pipeline system and associated methods described herein is the detection of people in images and / or videos, for example, people in stores, malls, and other establishments. By performing such detection, occupancy counts can be obtained. Since the generic object detection block 712 contains a person class, the generic object detection block 712 can be utilized to detect people.

[0082] Another example use of the computer vision pipeline system and associated methods described herein is the detection of insects on plants in images and / or videos, such as whiteflies on leaves. The tiled model block 710 can be used to split a high-resolution image of a plant into multiple images, detect insects in each image, and then recombine the images to aggregate the detection results.

[0083] Another use case of the computer vision pipeline system and associated methods described herein is the detection of license plates, such as U.S. license plates. A two-stage block can be utilized to detect license plates in an image and / or video. The first stage can use a detection model to crop a region of interest (ROI) in an image. The second stage can be performed on the cropped image. For example, a two-stage license plate detection block can first use a license plate detection model to crop a ROI in an image, and then a model to detect characters on the license plate can be performed on the cropped image. This second stage can be modeled as a character detection problem followed by algorithmic reconstruction, or as a recurrent neural network (RNN) based model.

[0084] Another use case of the computer vision pipeline system and related methods described herein is the quantification of plant growth for different plant species. There may be a standardized plant growth environment with a fixed camera viewpoint. The image crop block 704 can be used to crop the image of the ROI. The single model block 708 can then be executed to detect plants in the cropped image and quantify the growth of said plants.

[0085] Another example use of the computer vision pipeline system and associated methods described herein is the detection of chess positions. The Single Model block 708 can detect a chess board in an image, and then the Image Crop block 704 can crop the image to show only the chess board. The Image Resize block 706 can resize the image to reduce the resolution of the image. The resized image may be used for training or inference.

[0086] Another use case of the computer vision pipeline system and associated methods described herein is food detection. A two-stage model may be utilized to detect plates or bowls and then individual food items. It will be appreciated that other use cases are within the scope of the present disclosure.

[0087] 16A and 16B respectively show an example output 1600 and an example output 1650 of a predicted block in some embodiments. The output 1600 and the output 1650 may each be in a JavaScript Object Notation (JSON) file format. In some embodiments, the JSON file may have the following convention: "image name".extension.JSON. In FIG. 16A, the output 1600 may be from the logo detection block 720. The output 1600 includes a rectangle type indicating a bounding box. The output includes child tags with data describing the detected logo. The data includes minimum and maximum XY coordinates. These minimum and maximum values ​​define the bounds of the rectangle. The values ​​of the XY coordinates are relative values ​​(0.0 to 1.0) for width and height, respectively. The score is a confidence score.

[0088] In FIG. 16B, output 1650 may be from the general object detection block 712. Output 1650 includes source information and output information. Source information includes information about the input image. Output information includes an identifier of the output image and / or video as a base64 encoded string if the user selected to include the output image and / or video when configuring the output destination or if an image transformation was performed on said image and / or video. Output information may also include a segmentation mask as a base64 encoded string. Output information also includes prediction data, including a confidence score and a path to the model used for detection.

[0089] 17 illustrates a block diagram of an exemplary digital device 1700 according to some embodiments. The digital device 1700 is shown in the form of a general-purpose computing device. The digital device 1700 includes at least one processor 1702, RAM 1704, a communication interface 1706, input / output devices 1708, storage 1710, and a system bus 1712 that couples various system components, including the storage 1710, to the at least one processor 1702. A system, such as a computing system, may be one or more of the digital devices 1700 or may include one or more of the digital devices 1700.

[0090] System bus 1712 represents any one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0091] Digital device 1700 typically includes a variety of computer system readable media, such as computer system readable storage media. Such media may be any available media accessible by any of the systems described herein and includes both volatile and nonvolatile media, removable and non-removable media.

[0092] In some embodiments, the at least one processor 1702 is configured to execute executable instructions (e.g., programs). In some embodiments, the at least one processor 1702 comprises circuitry or any processor capable of processing executable instructions.

[0093] In some embodiments, RAM 1704 stores programs and / or data. In various embodiments, working data is stored in RAM 1704. The data in RAM 1704 may be cleared or eventually transferred to storage 1710, such as before resetting and / or powering down digital device 1700.

[0094] In some embodiments, the digital device 1700 is coupled to a network, such as the communications network 112, via a communications interface 1706. Additionally, the user system 106, the computer vision pipeline system 104, the destination system 110, and / or the input source system 108 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet).

[0095] In some embodiments, input / output device 1708 may be any device that inputs data (e.g., a mouse, keyboard, stylus, sensor, etc.) or any device that outputs data (e.g., a speaker, a display, a virtual reality headset).

[0096] In some embodiments, storage 1710 may include computer system readable media in the form of non-volatile memory, such as read only memory (ROM), programmable read only memory (PROM), solid state drive (SSD), flash memory, and / or cache memory. Storage 1710 may further include removable / non-removable volatile / non-volatile computer system storage media. By way of example only, storage 1710 may provide for reading from and writing to non-removable non-volatile magnetic media. Storage 1710 may include one non-transitory computer readable medium, or multiple non-transitory computer readable media, which store programs or applications to perform functions such as those described herein with reference to FIG. 2, for example. Although not shown, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media, may be provided. In such case, each of these may be connected to the system bus 1712 by one or more data media interfaces. As further illustrated and described below, the storage 1710 may include at least one program product that includes a set (e.g., at least one) of a plurality of program modules configured to perform the functions of an embodiment of the present invention. In some embodiments, the RAM 1704 is found within the storage 1710.

[0097] A program / utility having a set of program modules (at least one), such as the computer vision pipeline system 104, may be stored in the storage 1710, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networked environment. The program modules generally implement the functions and / or methodologies of the embodiments of the present invention described herein.

[0098] Although not shown, it is to be understood that other hardware and / or software components may be used with the digital device 1700. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, and the like.

[0099] In this specification, exemplary embodiments are described in detail with reference to the accompanying drawings. However, since the present disclosure can be implemented in various ways, it should not be construed as being limited to the embodiments disclosed in this specification. On the contrary, these embodiments are provided for a complete and complete understanding of the present disclosure and for a complete conveyance of the scope of the present disclosure.

[0100] It will be appreciated that aspects of one or more embodiments may be embodied as a system, method, or computer program product. Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Further, aspects may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied therein.

[0101] Any combination of one or more computer readable media may be utilized. A computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage media include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a solid-state drive (SSD), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program or data used by or in connection with an instruction execution system, apparatus, or device.

[0102] The transitory computer-readable signal medium may include a propagated data signal in which the computer-readable program code is embodied, for example in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, an electromagnetic wave signal, an optical signal, or any suitable combination thereof.

[0103] The program code embodied in the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0104] Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Python, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program code may be fully executed in any of the systems described herein, or any combination of the systems described herein.

[0105] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, cause means for implementing the specified functions / acts in one or more blocks of the flowchart illustrations and / or block diagrams.

[0106] These computer program instructions may be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the instructions stored on the computer-readable medium result in an article of manufacture that includes instructions that implement the specified functions / acts in the flowchart illustrations and / or block diagrams.

[0107] The computer program instructions may be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable data processing apparatus, or other device to execute a series of operational steps to generate a computer-implemented process, such that the instructions, when executed by a processor of the computer or other programmable data processing apparatus, result in a process for implementing the specified functions / operations in one or more blocks of the flowchart diagrams and / or block diagrams.

[0108] Although specific examples are described above for illustrative purposes, various equivalent modifications are possible. For example, while processes or blocks are presented in a given order, alternative implementations may perform routines having steps in a different order or employ systems having blocks in a different order, and some processes or blocks may be removed, moved, added, subdivided, combined, and / or modified to provide alternative combinations or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks may be shown as being performed sequentially, these processes or blocks may instead be performed or implemented simultaneously or in parallel, or may be performed at different times. Additionally, any specific numbers described herein are merely examples, and alternative implementations may employ different values ​​or ranges.

[0109] Throughout this specification, a component, operation, or structure described as a single instance may be implemented by multiple instances. Multiple structures and functionalities presented as separate components in an example configuration may be implemented as one combined structure or component. Similarly, multiple structures and functionalities presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter herein. Additionally, any specific numbers described herein are merely examples, and alternative implementations may employ different values ​​or ranges.

[0110] Components may be described or illustrated as being contained within or connected to other components. Such descriptions or illustrations are merely examples and other configurations may achieve the same or similar functionality. Components may be described or illustrated as being "coupled," "couplable," "operably coupled," "communicatively coupled," or the like with other components. Such descriptions or illustrations should be understood to indicate that such components may cooperate or interact with each other and may be in direct or indirect physical, electrical, or communication contact with each other.

[0111] Components may be described or illustrated as being "configured to," "adapted to," "operative to," "configurable to," "adaptable to," "operable to," etc. Such descriptions or illustrations should be understood to encompass components in both active and inactive or standby states, unless the context requires otherwise.

[0112] It will be apparent that various modifications may be made and other embodiments may be utilized without departing from the broad scope of the discussion herein, and therefore, these and other variations on the exemplary embodiments are intended to be covered by this disclosure.

Claims

1. A non-transitory computer-readable medium containing executable instructions, comprising: The executable instructions are executable by one or more processors to perform a method; The method comprises: receiving a request to create a computer vision pipeline for processing images and / or videos; receiving an image and / or video input source for the computer vision pipeline; displaying a plurality of blocks for selection, said plurality of blocks comprising: a plurality of image transformation blocks, each of which transforms an image and / or video to generate a transformed image and / or video; a plurality of prediction blocks, one of the prediction blocks detecting an object or a class of objects in an image and / or video to generate a detected object or class of objects; the steps of: receiving a selection of one of the image transformation blocks and configuration information for the image transformation block; adding the image transformation block to the computer vision pipeline; receiving a selection of one of the prediction blocks and configuration information of the prediction block; adding the prediction block to the computer vision pipeline; receiving an output of the computer vision pipeline; receiving a request to activate the computer vision pipeline; activating the computer vision pipeline; receiving an input image and / or video from the input source; transforming the input image and / or video using the image transformation block to generate a transformed image and / or video; detecting an object or class of objects in the transformed image and / or video using the predicted blocks to generate a detected object or class of objects; and sending the detected object or class of object to the output destination. A non-transitory computer readable medium comprising:

2. The method further includes displaying a user interface element onto which one of the blocks can be dragged and dropped; said step of receiving a selection of said image conversion block and configuration information of said image conversion block includes a step of receiving a drag and drop of said image conversion block onto said user interface element; 2. The non-transitory computer-readable medium of claim 1, wherein the step of receiving a selection of the predictive block and configuration information for the predictive block comprises receiving a drag and drop of the predictive block onto the user interface element.

3. The method further includes receiving a computer vision pipeline type, the computer vision pipeline type being one of a batch type or a streaming type; If the computer vision pipeline type is the batch type, the method further comprises receiving a processing schedule, wherein the step of receiving input images and / or video from the input source comprises receiving input images and / or video from the input source according to the processing schedule; 2. The non-transitory computer-readable medium of claim 1, wherein when the computer vision pipeline type is the streaming type, receiving input images and / or video from the input source comprises continuously receiving input images and / or video from the input source.

4. The non-transitory computer-readable medium of claim 1 , wherein the plurality of image transformation blocks include an image crop block that crops an image and / or video, and an image resize block that resizes an image and / or video.

5. 2. The non-transitory computer-readable medium of claim 1, wherein the plurality of prediction blocks includes a tiled model block that divides an image and / or video into a plurality of images for object or object class detection, detects an object or object class in the plurality of images, and recombines the plurality of images to generate a transformed image and / or video, and generates the transformed image and / or video and object or object class detection.

6. The plurality of predicted blocks include: a label detection block for detecting objects in an image and / or video and generating one or more label descriptions for the detected objects; a landmark detection block that detects landmarks in the image and / or video and generates descriptions of the detected landmarks; a logo detection block that detects logos in images and / or videos and returns a description of the detected logos; The non-transitory computer-readable medium of claim 1 , comprising:

7. 2. The non-transitory computer-readable medium of claim 1 , the method further comprising displaying an overview of the computer vision pipeline, the overview including the input sources, the image transform block, the prediction block, an order of the image transform block and the prediction block within the computer vision pipeline, and the output destination.

8. 1. A system including at least one processor and a memory containing instructions, The instructions, by the at least one processor: receiving a request to create a computer vision pipeline for processing images and / or videos; receiving an image and / or video input source for the computer vision pipeline; displaying a plurality of blocks for selection, said plurality of blocks comprising: a plurality of image transformation blocks, each of which transforms an image and / or video to generate a transformed image and / or video; a plurality of prediction blocks, one of the prediction blocks detecting an object or a class of objects in an image and / or video to generate a detected object or class of objects; Including; receiving a selection of one of the image transformation blocks and configuration information of the image transformation block; adding the image transformation block to the computer vision pipeline; receiving a selection of one of the prediction blocks and configuration information of the prediction block; adding the prediction block to the computer vision pipeline; receiving an output of the computer vision pipeline; receiving a request to activate the computer vision pipeline; activating the computer vision pipeline; receiving input images and / or video from said input source; transforming the input image and / or video using the image transformation block to generate a transformed image and / or video; detecting an object or class of objects in the transformed image and / or video using the prediction blocks to generate a detected object or class of objects; Sending the detected object or class of object to the destination. For,it is possible to run the,system.

9. The instructions are further executable by the at least one processor to display a user interface element over which one of the blocks can be dragged and dropped; the instructions executable by the at least one processor to receive a selection of the image transformation block and configuration information for the image transformation block include instructions executable by the at least one processor to receive a drag and drop of the image transformation block onto the user interface element; The system of claim 8, wherein the instructions executable by the at least one processor to receive a selection of the predictive block and configuration information for the predictive block include instructions executable by the at least one processor to receive a drag and drop of the predictive block onto the user interface element.

10. The instructions are further executable by the at least one processor to receive a computer vision pipeline type, where the computer vision pipeline type is one of a batch type or a streaming type; If the computer vision pipeline type is the batch type, the instructions further comprise instructions executable by the at least one processor to receive a processing schedule, the instructions executable by the at least one processor to receive input images and / or video from the input source comprising instructions executable by the at least one processor to receive input images and / or video from the input source in accordance with the processing schedule; 9. The system of claim 8, wherein when the computer vision pipeline type is the streaming type, the instructions executable by the at least one processor to receive input images and / or video from the input source include instructions executable by the at least one processor to continuously receive input images and / or video from the input source.

11. The system of claim 8 , wherein the plurality of image transformation blocks includes an image crop block for cropping an image and / or video, and an image resize block for resizing an image and / or video.

12. 10. The system of claim 8, wherein the plurality of prediction blocks includes a tiled model block that divides an image and / or video into a plurality of images for object or object class detection, detects an object or object class in the plurality of images, and recombines the plurality of images to generate a transformed image and / or video, and generates the transformed image and / or video and the object or object class detection.

13. 10. The system of claim 8, wherein the plurality of prediction blocks includes a tiled model block that divides an image and / or video into a plurality of images for object or object class detection, detects an object or object class in the plurality of images, and recombines the plurality of images to generate a transformed image and / or video, and generates the transformed image and / or video and the object or object class detection.

14. 9. The system of claim 8, wherein the instructions are further executable by the at least one processor to display an overview of the computer vision pipeline, the overview including the input sources, the image transform block, the prediction block, an order of the image transform block and the prediction block within the computer vision pipeline, and the output destination.

15. receiving a request to create a computer vision pipeline for processing images and / or videos; receiving an image and / or video input source for the computer vision pipeline; displaying a plurality of blocks for selection, said plurality of blocks comprising: a plurality of image transformation blocks, each of which transforms an image and / or video to generate a transformed image and / or video; a plurality of prediction blocks, one of the prediction blocks detecting an object or a class of objects in an image and / or video to generate a detected object or class of objects; the steps of: receiving a selection of one of the image transformation blocks and configuration information for the image transformation block; adding the image transformation block to the computer vision pipeline; receiving a selection of one of the prediction blocks and configuration information of the prediction block; adding the prediction block to the computer vision pipeline; receiving an output of the computer vision pipeline; receiving a request to activate the computer vision pipeline; activating the computer vision pipeline; receiving an input image and / or video from the input source; transforming the input image and / or video using the image transformation block to generate a transformed image and / or video; detecting an object or class of objects in the transformed image and / or video using the predicted blocks to generate a detected object or class of objects; and sending the detected object or class of object to the output destination. A method comprising:

16. displaying a user interface element onto which one of the blocks can be dragged and dropped; said step of receiving the selection of the image conversion block and the configuration information of the image conversion block includes the step of receiving a drag and drop of the image conversion block onto the user interface element; The method of claim 15 , wherein the step of receiving a selection of the predictive block and configuration information for the predictive block comprises receiving a drag and drop of the predictive block onto the user interface element.

17. receiving a computer vision pipeline type, the computer vision pipeline type being one of a batch type or a streaming type; if the computer vision pipeline type is the batch type, further comprising receiving a processing schedule, wherein the step of receiving input images and / or video from the input source comprises receiving input images and / or video from the input source according to the processing schedule; 16. The method of claim 15, wherein when the computer vision pipeline type is the streaming type, receiving input images and / or video from the input source comprises continuously receiving input images and / or video from the input source.

18. The method of claim 15 , wherein the plurality of image transformation blocks includes an image crop block for cropping an image and / or video, and an image resize block for resizing an image and / or video.

19. 16. The method of claim 15, wherein the plurality of prediction blocks includes a tiled model block that divides an image and / or video into a plurality of images for object or object class detection, detects an object or object class in the plurality of images, and recombines the plurality of images to generate a transformed image and / or video, and generates the transformed image and / or video and the object or object class detection.

20. The plurality of predicted blocks include: a label detection block for detecting objects in an image and / or video and generating one or more label descriptions for the detected objects; a landmark detection block that detects landmarks in the image and / or video and generates descriptions of the detected landmarks; a logo detection block that detects logos in images and / or videos and returns a description of the detected logos; 16. The method of claim 15, comprising:

Images