Parametric curve based detector network

Abstract

Embodiments can provide improved techniques for object detection to improve the accuracy of object discovery and boundary prediction using a parametric curve defined by a plurality of control points. For example, in one embodiment, a method implemented in a computer system that includes a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method can include receiving an image, extracting a plurality of features related to an object shown in the image from the image, generating at least one plurality of points representing a parametric curve that defines the object shown in the image from the extracted features, and outputting the plurality of points representing the parametric curve.

Description

Background Technology

[0001] This invention relates to techniques for object detection, in order to improve the accuracy of object discovery and boundary prediction using parametric curves defined by multiple control points.

[0002] Machine learning-based object detectors use AABB (axis-aligned bounding boxes) to provide object predictions. On the other hand, machine learning-based image segmentation models provide pixel-level (or near-pixel) class predictions. A major drawback of such detectors is that AABB does not capture the shape of many real-world objects very well. The main disadvantages of segmentation models are: a) pixel-level objects are not always well-defined because boundaries are not always clear (e.g., in medical applications), and a single pixel can “belong” to multiple classes because they are mixed together; and more importantly, b) it is difficult to express object importance because in some applications, adding 5 correct pixels to an already discovered object is significantly less important than finding 5 correct pixels of a new, yet-to-be-discovered object (this is referred to as “object importance”).

[0003] Object detection performance can be measured by a combination of: 1) what objects are found, and 2) how tight and accurate the boundary predictions are. Typically, these two parameters are combined by defining object shape matching criteria, comparing the predictions to the ground truth boundaries of the actual objects, and considering a miss if the similarity of the comparison is below a certain threshold.

[0004] Therefore, improved techniques for object detection are needed to improve the accuracy of object discovery and boundary prediction. Summary of the Invention

[0005] The embodiments provide improved techniques for object detection to improve the accuracy of object discovery and boundary prediction using parametric curves defined by multiple control points. For example, the embodiments can provide significantly tighter boundary predictions for objects while accurately representing object importance. Instead of predicting four values ​​per object (representing fine-tuning of the AABB position), the embodiments can predict K curve control points instead. This achieves a better balance between discovering objects and describing their shapes, leading to superior object detection performance. Furthermore, the embodiments can provide object shape regularization. Moreover, the ability to represent the boundaries of various shapes with a small number of control points is superior to techniques that attempt to predict polygonal shapes, as polygons can contain many vertices and the model is more likely to overshoot.

[0006] For example, in one embodiment, a method may be implemented in a computer system including a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor. The method may include: receiving an image; extracting a plurality of features from the image related to an object shown in the image; generating from the extracted features at least one plurality of points representing a parametric curve defining the object shown in the image; and outputting the plurality of points representing the parametric curve.

[0007] In an embodiment, a regression model can be used to extract multiple features. The method may include optimizing the matching between a baseline truth object and a predicted anchor using a loss function. This loss function may include one of the following curve similarity measures, computed in the curve space of the curve or at a selected finite resolution: at least one of Fréchet distance, Hausdorff distance, Bottleneck distance, and a Lie algebra-based measure. The method may include optimizing the weights of the regression model using a curve distance metric. The generation may include generating multiple points representing multiple parametric curves that define objects shown in the image. These multiple parametric curves may be extended to overlap and cover the entire received image.

[0008] In one embodiment, a system may include a processor, a memory accessible to the processor, and computer program instructions stored in the memory and executable by the processor to perform the following operations: receiving an image; extracting a plurality of features from the image related to an object shown in the image; generating from the extracted features at least one or more points representing a parametric curve defining the object shown in the image; and outputting the plurality of points representing the parametric curve.

[0009] In one embodiment, a computer program product may include a non-transitory computer-readable storage having program instructions embodied therein, the program instructions being executable by a computer to cause the computer to perform a method comprising: receiving an image; extracting from the image a plurality of features relating to an object shown in the image; generating from the extracted features at least one plurality of points representing a parametric curve defining the object shown in the image; and outputting the plurality of points representing the parametric curve. Attached Figure Description

[0010] The details of the invention regarding its structure and operation can be best understood by referring to the accompanying drawings, in which the same reference numerals and names refer to the same elements.

[0011] Figure 1An exemplary embodiment of a neural network architecture according to an embodiment of the present technology is shown.

[0012] Figure 2 This is an exemplary flowchart of the object detection process in a neural network architecture according to an embodiment of the present technology.

[0013] Figure 3 This is an exemplary illustration of anchors and control points for a parametric curve predicted according to the definition of an embodiment of the present technology.

[0014] Figure 4 This is an exemplary block diagram of a computer system in which the processes described in the embodiments herein can be implemented. Detailed Implementation

[0015] The embodiments provide improved techniques for object detection to improve the accuracy of object discovery and boundary prediction using parametric curves defined by multiple control points. For example, the embodiments can provide significantly tighter boundary predictions for objects while accurately representing object importance. Instead of predicting four values ​​per object (representing fine-tuning of the AABB position), the embodiments can predict K curve control points instead. This achieves a better balance between discovering objects and describing their shapes, leading to superior object detection performance. Furthermore, the embodiments can provide object shape regularization. Moreover, the ability to represent the boundaries of various shapes with a small number of control points is superior to techniques that attempt to predict polygonal shapes, as polygons can contain many vertices and the model is more likely to overshoot.

[0016] An exemplary embodiment of the neural network architecture 100 is in Figure 1 As shown in the image. It is best to combine... Figure 2 Let's check them together. Figure 2This is an exemplary flowchart of the object detection process 200 in a neural network architecture 100. In this example, architecture 100 may include a feature extraction submodule 102, at least one regression head 104, a defined loss function 106, an optimizer 108, and regularization 110. Process 200 begins at 202, where the feature extraction submodule 103 can extract relevant features from each input image 112, such as features related to the objects shown in the image, referred to as "main features (MAIN_FEATURES)". At 204, the regression model head 104 can take the extracted main features as input and can output points, such as N anchor points x C control points, which represent one or more parametric curves that potentially define objects in the current image. At 206, the loss function 106 can be applied. The loss function 106 can be a defined loss function (or sub-term) that attempts to optimize the match between the baseline truth object and the predicted anchors. At 208, the optimizer 108 can optimize the model weights according to the defined loss function. The optimizer 108 can use any curve distance metric, such as Friesian distance, Hausdorff distance, bottleneck distance, Lie algebra-based metrics, etc. It can also use a selected finite resolution of the curve, thus allowing the use of polygon-based metrics as well. (Note that this is not equivalent to the network directly outputting polygons.) At 210, optional regularization terms 110 can influence the shape characteristics of the predicted parametric curves. Such characteristics can include smoothness, convexity, etc. At 212, predicted output points 112 representing one or more parametric curves and potential boundary objects in the current image can be output.

[0017] Figure 3 An example of anchors and control points defining the parametric curves for prediction is shown as output from process 200. In this example, each anchor representing a potential detection can begin with the default 302. Control points are part of the model's anchor-by-anchor prediction, allowing it to express more complex shapes using very few numerical values. For example, anchor 302 is simply the default anchor. Other anchors can have more complex shapes with the same number of control points, such as anchor 304. Process 200 can predict anchor 306, which is expanded to overlap and cover the entire input image. Optimizer 108 and / or regularization term 110 can modify the shape 304 of one or more anchors from the default shape 302.

[0018] Figure 4An exemplary block diagram of a computer system 400 is shown, in which the processes and components described in the embodiments herein can be implemented. The computer system 400 can be implemented using one or more programmable general-purpose computer systems, such as embedded processors, system-on-a-chip, personal computers, workstations, server systems, and minicomputers or mainframes, or implemented in a distributed networked computing environment. The computer system 400 may include one or more processors (CPUs) 402A-402N, input / output circuitry 404, a network adapter 406, and memory 408. The CPUs 402A-402N execute program instructions to perform the functions of this communication system and method. Typically, the CPUs 402A-402N are one or more microprocessors, such as Intel... processor. Figure 4 An embodiment of the computer system 400 is shown, implemented as a single multiprocessor computer system, wherein multiple processors 402A-402N share system resources, such as memory 408, input / output circuitry 404, and network adapter 406. However, the communication system and method also include embodiments in which the computer system 400 is implemented as multiple networked computer systems, which may be single-processor computer systems, multiprocessor computer systems, or combinations thereof.

[0019] Input / output circuitry 404 provides the ability to input data to or output data to computer system 400. For example, input / output circuitry may include input devices (such as keyboards, mice, touchpads, trackballs, scanners, analog-to-digital converters, etc.), output devices (such as video adapters, monitors, printers, etc.), and input / output devices (such as modems, etc.). Network adapter 406 interfaces device 400 with network 410. Network 410 may be any public or private LAN or WAN, including but not limited to the Internet.

[0020] Memory 408 stores program instructions executed by CPU 402 and data used and processed by CPU 402 to perform the functions of computer system 400. Memory 408 may include, for example, electronic memory devices such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc., and electromechanical memory that can use an Integrated Drive Electronics (IDE) interface or its variants or enhancements, such as Enhanced IDE (EIDE) or Ultra Direct Memory Access (UDMA), such as disk drives, tape drives, optical disc drives, etc., or an interface based on Small Computer System Interface (SCSI) or its variants or enhancements, such as Fast SCSI, Wide SCSI, Fast and Wide SCSI, etc., or Serial Advanced Technology Accessory (SATA) or its variants or enhancements, or Fibre Channel Arbitrated Loop (FC-AL) interface.

[0021] The contents of memory 408 can be changed according to the functions that computer system 400 is programmed to perform. Figure 4 The example shown illustrates exemplary memory contents of routines and data representing embodiments of the above-described process. However, those skilled in the art will recognize that these routines, along with the memory contents associated with them, may not be contained within a single system or device, but may be distributed across multiple systems or devices based on well-known engineering considerations. This system and method may include any and all such arrangements.

[0022] exist Figure 4 In the example shown, memory 408 may include a third utterance generation routine 412, a model routine 414, a bot routine 416, training data 418, and an operating system 420. As described above, the third utterance generation routine 412 may include software for generating data (such as rich training data and third utterance generation). The model routine 414 may include software providing text-to-text modeling capabilities, as described above. The bot routine 416 may include software for implementing an automated dialogue system, as described above. The operating system 418 can provide overall system functionality.

[0023] like Figure 4As shown, this communication system and method can be implemented on one or more systems providing multiprocessor, multitasking, multiprocessing, and / or multithreaded computing, as well as on systems providing only single-processor, single-threaded computing. Multiprocessor computing involves performing computation using more than one processor. Multitasking computing involves performing computation using more than one operating system task. A task is an operating system concept, referring to a combination of a program being executed and bookkeeping information used by the operating system. Whenever a program is executed, the operating system creates a new task for it. This task is like an envelope for the program, as it identifies the program with a task number and attaches other bookkeeping information to it. This includes Linux, and Many operating systems are capable of running many tasks simultaneously and are known as multitasking operating systems. Multitasking is the ability of an operating system to execute more than one executable file at a time. Each executable file runs in its own address space, meaning that executable files cannot share any memory in their own memory. This has advantages because it is impossible for any program to disrupt the execution of any other program running on the system. However, programs cannot exchange any information except through the operating system (or by reading files stored on the file system). Multiprocess computing is similar to multitasking computing because the terms task and process are often used interchangeably, although some operating systems make a distinction between the two.

[0024] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. A computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention. The computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution apparatus.

[0025] Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0026] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0027] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by personalizing the electronic circuitry with state information from the computer-readable program instructions in order to perform aspects of this invention.

[0028] The present invention is described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0029] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0030] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0031] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0032] While specific embodiments of the invention have been described, those skilled in the art will understand that other embodiments equivalent to the described embodiments exist. Therefore, it should be understood that the invention is not limited to the embodiments specifically shown, but only to the scope of the appended claims.

Claims

1. A method implemented in a computer system, the computer system comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method comprising: Receive image; Extract features related to the objects shown in the image from the image; An initial prediction anchor is generated from the extracted features and the default anchor, the initial prediction anchor and the default anchor having the same number of control points; The matching between the baseline truth of the object shown in the image and the initial prediction anchor is optimized using a loss function to obtain a parametric curve defining the object shown in the image, represented by multiple control points of the final prediction anchor; and The output represents the plurality of control points of the parameter curve.

2. The method of claim 1, wherein a regression model is used to perform the feature extraction.

3. The method according to claim 2, wherein, The loss function includes one of the curve similarity measures computed in the curve space of the parametric curve or at a selected finite resolution, the curve similarity measure including at least one of Fraser distance, Hausdorff distance, bottleneck distance, and Lie algebra-based measure.

4. The method of claim 2, further comprising: The weights of the regression model are optimized using a curve distance metric.

5. The method according to claim 1, wherein, The parameter curves are expanded to overlap and cover the entire received image.

6. A system for object detection, comprising a processor, a memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform the following operations: Receive image; Extract features related to the objects shown in the image from the image; An initial prediction anchor is generated from the extracted features and the default anchor, the initial prediction anchor and the default anchor having the same number of control points; The matching between the baseline truth of the object shown in the image and the initial prediction anchor is optimized using a loss function to obtain a parametric curve defining the object shown in the image, represented by multiple control points of the final prediction anchor; and The output represents the plurality of control points of the parameter curve.

7. The system of claim 6, wherein a regression model is used to perform the extraction of the features.

8. The system according to claim 7, wherein, The loss function includes one of the curve similarity measures computed in the curve space of the parametric curve or at a selected finite resolution, the curve similarity measure including at least one of Fraser distance, Hausdorff distance, bottleneck distance, and Lie algebra-based measure.

9. The system according to claim 7, further comprising: The weights of the regression model are optimized using a curve distance metric.

10. The system according to claim 6, wherein, The parameter curves are expanded to overlap and cover the entire received image.

11. A computer program product comprising program instructions executable by a computer to cause the computer to perform a method comprising: Receive image; Extract features related to the objects shown in the image from the image; An initial prediction anchor is generated from the extracted features and the default anchor, the initial prediction anchor and the default anchor having the same number of control points; The matching between the baseline truth of the object shown in the image and the initial prediction anchor is optimized using a loss function to obtain a parametric curve defining the object shown in the image, represented by multiple control points of the final prediction anchor; and The output represents the plurality of control points of the parameter curve.

12. The computer program product of claim 11, wherein a regression model is used to perform the extraction of the features.

13. The computer program product according to claim 12, wherein, The loss function includes one of the curve similarity measures computed in the curve space of the parametric curve or at a selected finite resolution, the curve similarity measure including at least one of Fraser distance, Hausdorff distance, bottleneck distance, and Lie algebra-based measure.

14. The computer program product of claim 12, wherein the parameter curve is expanded to overlap and cover the entire received image.

Images