Computer software module device, circuit module device, device and method for improving object detection by compensating for confidence determination of detected objects

Abstract

An object detection device (100) includes a controller (101) configured to detect objects using a multi-scale convolutional neural network, wherein the controller (101) is further configured to: receive (312, 410) image data representing an image (10) including an object (11) to be detected located at a distance (d) to the image (10); classify (314, 430) whether the object (11) to be detected is at risk of being incorrectly detected based on the distance (d); and if so, compensate (315, 440) the object detection by adapting (316) object detection parameters.

Description

Technical Field

[0001] This invention relates to an arrangement of devices, including computer software modules, including circuits, apparatus, and methods for providing an improved object detection method, and particularly to devices, including computer software modules, including circuits, and methods for providing an improved object detection method through improved object detection. Background Technology

[0002] Object detection is a computer science technique related to computer vision and object detection, and it involves detecting instances of a certain class of semantic objects (such as people, buildings, or cars) in digital images and videos. Good research areas in object detection include face detection and pedestrian detection. Object detection has applications in many areas of computer vision, including image retrieval and video surveillance. Methods for object detection typically fall into either machine learning-based (such as deep learning) or classical approaches. Deep learning techniques are capable of end-to-end object detection (without requiring specific feature definition as in machine learning methods) and are typically based on convolutional neural networks (CNNs), such as SSD (Single Shot Detector) or Region Proposal R-CNN, or variants such as Fast R-CNN, Faster R-CNN, or Cascaded R-CNN.

[0003] In object detection using CNNs, the so-called multi-scale CNN has become commonplace. A multi-scale CNN is essentially a collection or group of CNN models with varying input sizes, where the outputs of those parallel layers are later combined for the identification or detection of relevant objects. Multi-scale CNNs are particularly useful for detecting objects of different sizes in images (or video streams). A typical implementation of such a multi-scale CNN involves scaling down (and later scaling up) the data in multiple steps. This scaling down (and scaling up) is typically done by a factor of two for each step (or multiple steps). This type of architecture has proven very useful for detecting features of different sizes and has therefore become frequently used and widely adopted.

[0004] However, after profound and creative reasoning and experimental research, the inventors have recognized the existence of problems that were previously unknown, and have also identified these previously unknown problems. Therefore, the inventors also recognize the need to solve such problems. Summary of the Invention

[0005] As stated above, the inventors have recognized and identified previously unknown problems that have not yet been detected. Therefore, the mere recognition of this problem is itself inventive. Since object detection is utilized in many aspects of modern life, it is necessary to provide solutions that the inventors have also provided, which are themselves inventive. Furthermore, the simplicity and ingenuity of the provided solutions are also inventive.

[0006] Figure 1A Example image 10 is shown, in which two types of objects 11 are found: two cars 11A and 11B and three people 11C, 11D, and 11E. The cars and people are merely examples of objects, and it should be noted that the issues discussed herein are not limited to any particular type of object, but are equally applicable to all types of objects. The objects 11 in example image 10 are: a first car 11A at a first horizontal distance HD1 in image 10; a second, slightly larger car 11B at a second horizontal distance HD2 in image 10; a first person 11C at a first vertical distance VD1 in image 10; a second, slightly smaller person 11D at the same first vertical distance VD1 in image 10; and a third person 11E having the same size as the second person 11D but at a second vertical distance VD2 in image 10. In this example, horizontal distances are indicated from left to right, and vertical distances are indicated from top to bottom; however, it should be noted that these distances can also be measured in the opposite directions, and these distances can be measured in different ways.

[0007] As can be seen and as stated, objects have different sizes, and therefore multi-scale CNNs will most commonly be used for object detection. It should be noted that multi-scale CNNs can be used to detect objects regardless of size differences (if any). It should also be noted that the actual size is irrelevant to the problem being identified, and the problem will apply equally to large and small objects.

[0008] Through profound and creative reasoning, the inventors have recognized that errors or low confidence levels when detecting objects are, in some cases, related to the location of the object within an image, as certain locations will provide lower confidence for object detection. Through further profound and creative reasoning, the inventors have further recognized that these locations depend on the shape or extent of the object being detected. And, even more specifically, through profound and creative reasoning, the inventors have recognized that objects suffer lower confidence at locations that affect the interpretation of their corresponding extent. This means that objects 11 with basic horizontal extent (such as cars 11A and 11B) will primarily suffer lower confidence (when detecting objects) at certain horizontal distances HD, and objects 11 with basic vertical extent (such as people 11C, 11D, and 11E) will primarily suffer lower confidence at certain vertical distances.

[0009] However, since the other extension would also be negatively affected at the corresponding distance, the inventors have also provided compensation for both vertical and horizontal extension (if at a certain horizontal and / or vertical distance).

[0010] For the context described in this article, horizontal stretch will be defined as width (in...) Figure 1A (represented by W) is greater than the height (in) Figure 1A The extension (denoted by H) is defined as the width being less than the height, while the vertical extension will be defined as the extension where the width is less than the height. In one interpretation, the width will be increased / decreased by a factor greater than 1 so that the extension of an object is defined as vertical or horizontal. Examples of these factors are 1, 1.1, 1.25, 1.5, and 2.

[0011] As described above, in some embodiments, both (or any one of them) stretching can be compensated.

[0012] Therefore, the inventors not only recognized the existence of the problem, but they also recognized its details. It should be noted that because CNNs are trained on such a massive number of objects with incredible diversity, so many factors are at play that is truly an impressive achievement to isolate some of these factors and contribute them to the problem.

[0013] The inventors also confirmed and verified the problems they had identified through experimental research.

[0014] As mentioned above, the inventors also recognized that the primary contributing factor to the problem was the object's position, not its size. However, as will be discussed below, the object's size does indeed play a role.

[0015] Furthermore, the inventors recognized the cause of the problem: scaling performed in multi-scale CNNs.

[0016] Figure 1B A series of plots 21, 22, and 23 show the relationship between the confidence value C for object 11 and their distances (vertical VD and horizontal HD) to image 10. The two plots above, or plots 21 and 22, show the relationship between the confidence value C for object 11 and their distances to image 10 (vertical VD and horizontal HD). Figure 1A The confidence value C of cars 11A and 11B in the image, where the distance d is the distance to the image ( Figure 1A The horizontal distance HD is referenced as 10. The curve 21 above is for smaller objects (such as...). Figure 1A The small car 11A in the middle, while the middle curve 22 is for larger objects (such as...). Figure 1A The large car 11B in the middle). The bottom curve 23 shows the vertical object (such as) drawn for distance d. Figure 1A The confidence level C of people 11C, 11D, 11E), where the distance d is to the image (in Figure 1A The vertical distance VD is referenced in (10). As can be seen from curves 21, 22, and 23, the confidence C drops or is reduced (at regular intervals for the distance to the image). In this example, there is a dip or through in the confidence C at every 16 pixels. The inventors have also identified that for particularly small objects (such as those falling below a size threshold), there is an additional dip between each dip, or more precisely, a dip in the confidence C at every 8 pixels of distance. Therefore, the confidence drop occurs at the confidence distance to the image (in... Figure 1B The confidence distance is referenced at (cd) or near that confidence distance, regardless of the actual size. The only effect of the size is whether the object falls below or above the threshold size value, in which case the confidence distance is doubled (if it falls above the threshold).

[0017] It should also be noted that the confidence distance is the same for both the vertical and horizontal distances to the image, and this distance will therefore be referred to below as the object-specific distance, where the direction of the distance is determined by the object's classification (vertical or horizontal).

[0018] It should be noted that the confidence distance (cd) values ​​are merely examples and can vary depending on the properties of the CNN used and the scaling factor employed. However, they are suitable for the most commonly used CNNs with a scaling factor of 2.

[0019] It should also be noted that when using different scaling factors, the confidence distance cd can be different for vertical and horizontal distances.

[0020] Confidence distances can be based on experiments, where they are essentially measurements. Confidence distances can also be derived from CNN properties such as the scale factor as described above.

[0021] Therefore, one purpose of this teaching is to overcome or at least reduce or alleviate the above references. Figure 1A and 1B The issues under discussion.

[0022] According to one aspect, an object detection device is provided, the object detection device including a controller configured to detect objects using a multi-scale convolutional neural network, wherein the controller is further configured to: receive image data representing an image, the image including an object to be detected at a distance (d) to the image; classify, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, adapt the object detection accordingly.

[0023] In one embodiment, the controller is further configured to classify whether an object to be detected is at risk of being incorrectly detected based on distance (d) by determining whether the distance (d) is located at a multiple of the confidence distance (cd) plus a distance constant.

[0024] In one embodiment, the controller is further configured to classify whether an object to be detected is at risk of being incorrectly detected based on distance (d) by determining whether the distance (d) is within a margin of multiples of the confidence distance (cd).

[0025] In one embodiment, the controller is further configured to classify whether an object to be detected is at risk of being incorrectly detected based on distance (d) by determining whether the object's size is above a size threshold, in which case the multiple to be multiplied by the confidence distance (cd) is an even multiple.

[0026] In one embodiment, the controller is further configured to classify whether the object to be detected is at risk of being incorrectly detected based on distance (d) by determining whether the object to be detected is a vertical object (where distance (d) is the vertical distance (VD) and confidence distance represents the vertical distance) and / or whether the object to be detected is a horizontal object (where distance (d) is the horizontal distance (HD) and confidence distance represents the horizontal distance).

[0027] In one embodiment, the controller is further configured to classify whether the object to be detected is at risk of being incorrectly detected based on distance (d) by first determining that the provided confidence value (C) is within a confidence interval.

[0028] In one embodiment, the confidence interval is a range below the class threshold used to determine whether an object to be detected belongs to the corresponding class.

[0029] In one embodiment, the confidence distance is based on the scale scaling factor of a multi-scale CNN.

[0030] According to one aspect, a method for object detection using a multi-scale CNN in an object detection device is provided, wherein the method includes: receiving image data representing an image, the image containing an object to be detected, the object being located at a distance (d) in the image; classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, adapting the object detection accordingly.

[0031] According to one aspect, a software module device is provided for object detection using a multi-scale CNN in an object detection apparatus, wherein the software module device includes: a software module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a software module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a software module for adapting the object detection accordingly.

[0032] According to one aspect, an object detection apparatus is provided including a circuit module for object detection using a multi-scale CNN, comprising: a circuit module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a circuit module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a circuit module for adapting the object detection accordingly.

[0033] According to one aspect, an object detection device is provided, comprising a controller configured to detect objects using a multi-scale convolutional neural network, wherein the controller is further configured to: receive image data representing an image, the image including an object to be detected located at a distance (d) in the image; classify, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, compensate for the object detection by adapting object detection parameters.

[0034] In one embodiment, the controller is configured to adapt object detection parameters by lowering the class threshold.

[0035] In one embodiment, the controller is configured to adapt the object detection parameters by increasing the confidence (C) for the object.

[0036] In one embodiment, the controller is configured to adapt object detection parameters by lowering the class threshold and by increasing the confidence (C) for the object.

[0037] In one embodiment, the controller is configured to reduce the class threshold by a certain amount based on the distance between the object and a multiple of the confidence distance (cd).

[0038] In one embodiment, the controller is configured to increase the confidence (C) of the object by a certain amount based on a distance that is a multiple of the confidence distance (cd).

[0039] In one embodiment, the controller is configured to classify, during compensated detection, whether an object to be detected is at risk of being incorrectly detected based on distance (d).

[0040] In one embodiment, the controller is configured to retrieve parameters to be used via a table lookup, classify whether an object to be detected is at risk of being incorrectly detected based on distance (d), and adapt object detection parameters, wherein the lookup table carries parameters to be used on the object based on distance (d).

[0041] In one embodiment, the object detection device further includes a memory enabled to store the lookup table.

[0042] In one embodiment, the object detection device further includes a communication interface for retrieving information from the lookup table.

[0043] According to one aspect, a method for object detection using a multi-scale CNN in an object detection device is provided, wherein the method includes: receiving image data representing an image including an object to be detected located at a distance (d) in the image; classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, compensating for object detection by adapting object detection parameters.

[0044] According to one aspect, a software module device is provided for object detection using a multi-scale CNN in an object detection apparatus, wherein the software module device includes: a software module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a software module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a software module for compensating for object detection by adapting object detection parameters.

[0045] According to one aspect, an object detection apparatus is provided including a circuit module for object detection using a multi-scale CNN, comprising: a circuit module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a circuit module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a circuit module for compensating for object detection by adapting object detection parameters.

[0046] According to one aspect, an object detection apparatus is provided, including a controller configured to detect objects using a multi-scale convolutional neural network, wherein the controller is further configured to: receive image data representing an image, the image including an object to be detected located at a distance (d) in the image; classify, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, compensate for the object detection by shifting the image.

[0047] In one embodiment, the controller is further configured to shift the image by a shift distance (sd) greater than the margin distance (md) and less than the confidence distance (cd), wherein if the distance (d) is within the margin distance (md) which is a multiple of the confidence distance (cd), the object to be detected is at risk of being incorrectly detected.

[0048] In one embodiment, the controller is further configured to compensate for object detection by shifting the image, which involves performing a first object detection on the image and shifting the image, and performing a second object detection on the shifted image and comparing the results of the first and second object detections.

[0049] In one embodiment, object detection on an image provides a first confidence level (C1), and object detection on a shifted image provides a second confidence level (C1), wherein the controller is further configured to compare the results by comparing the first confidence level (C1) and the second confidence level (C2).

[0050] In one embodiment, the controller is further configured to classify whether the object to be detected is at risk of being incorrectly detected based on distance (d) by determining that a second confidence level (C2) is higher than a first confidence level (C1).

[0051] In one embodiment, the controller is configured to shift the image by shifting a subsequent image.

[0052] In one embodiment, the controller is configured to shift the image by shifting the image, providing a second instance of the image, and feeding the second instance to object detection.

[0053] According to one aspect, a method for object detection using a multi-scale CNN in an object detection device is provided, wherein the method includes: receiving image data representing an image including an object to be detected located at a distance (d) in the image; classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, compensating for object detection by shifting the image.

[0054] According to one aspect, a software module device is provided for object detection using a multi-scale CNN in an object detection apparatus, wherein the software module device includes: a software module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a software module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a software module for compensating for object detection by shifting the image.

[0055] According to one aspect, an object detection apparatus is provided including a circuit module for object detection using a multi-scale CNN, comprising: a circuit module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a circuit module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a circuit module for compensating for object detection by shifting the image.

[0056] According to one aspect, an object detection device is provided, including a controller configured to detect objects using a multi-scale convolutional neural network, wherein the controller is further configured to: receive image data representing an image, the image including an object to be detected located at a distance (d) in the image; classify, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, compensate for the object detection by adapting object detection parameters and by shifting the image.

[0057] In one embodiment, the controller is further configured to compensate for object detection by shifting the image, performing a first object detection on the image and shifting the image, performing a second object detection on the shifted image, and adapting object detection parameters based on the results of the first and second object detections.

[0058] In one embodiment, the controller is further configured to adapt object detection parameters based on the results of first and second object detections, wherein the object detection parameters are adapted based on the difference in confidence levels provided by the first and second object detections.

[0059] In one embodiment, the object detection device further includes a communication interface, wherein the controller is further configured to shift the image by causing a remote controller to shift the image via the communication interface and performing object detection on the shifted image.

[0060] In one embodiment, the controller is configured to shift the image to establish the adaptation.

[0061] In one embodiment, the controller is configured to shift the image to verify the adaptation.

[0062] In one embodiment, the controller is configured to shift the image to correct the adaptation.

[0063] According to one aspect, a method for object detection using a multi-scale CNN in an object detection device is provided, wherein the method includes: receiving image data representing an image including an object to be detected located at a distance (d) in the image; classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and if so, compensating for the object detection by adapting object detection parameters and by shifting the image.

[0064] According to one aspect, a software module device is provided for object detection using a multi-scale CNN in an object detection apparatus, wherein the software module device includes: a software module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a software module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a software module for compensating for object detection by adapting object detection parameters and by shifting the image.

[0065] According to one aspect, an object detection apparatus is provided including a circuit module for object detection using a multi-scale CNN, comprising: a circuit module for receiving image data representing an image, the image including an object to be detected located at a distance (d) in the image; a circuit module for classifying, based on the distance (d), whether the object to be detected is at risk of being incorrectly detected; and, if so, a circuit module for compensating for object detection by adapting object detection parameters and by shifting the image.

[0066] In any embodiment of the above aspects, the object detection device is a smartphone or tablet computer.

[0067] In one embodiment of any of the inventions described herein, the object detection device is an optical perspective device.

[0068] It should be noted that any, some, or all of the aspects discussed herein may be combined or used to complement each other, and embodiments discussed with respect to one aspect may also be applicable to other aspects, and therefore may be combined arbitrarily.

[0069] It should also be noted that in some embodiments, the adaptation is performed for both horizontal and vertical alignment.

[0070] According to one aspect, a computer-readable medium carrying computer instructions is provided, which, when loaded into and executed by a controller of an object detection device, enable the object detection device to implement any of the methods described herein.

[0071] This solution can be implemented as a software solution, a hardware solution, or a hybrid of software and hardware components.

[0072] Further embodiments and advantages of the invention will be set forth in a detailed description. It should be noted that the teachings herein find application in digital imaging devices in digital photography and in many fields of computer vision, including image retrieval, robot vision, augmented reality, and video surveillance. Attached Figure Description

[0073] Embodiments of the invention will now be described with reference to the accompanying drawings, which illustrate non-limiting examples of how the inventive concepts can be put into practice.

[0074] Figure 1A A schematic view of an image including several objects is shown;

[0075] Figure 1B This shows a view of three curves (plots) plotted according to the prior art for the confidence values ​​when detecting objects based on the distance to the image;

[0076] Figure 2A A schematic view of an object detection device according to an embodiment of the present invention is shown;

[0077] Figure 2B A schematic view of an object detection device according to an embodiment of the present invention is shown;

[0078] Figure 2C A schematic view of an object detection device according to an embodiment of the present invention is shown;

[0079] Figure 3A A schematic view of a general object detection model according to an embodiment of the teachings herein is shown;

[0080] Figure 3B A schematic view of an alternative object detection model according to one embodiment of the teachings herein is shown;

[0081] Figure 3C A schematic view of an alternative object detection model according to one embodiment of the teachings herein is shown;

[0082] Figure 3D A schematic view of a compensated-detection shift image is shown according to one embodiment of the teachings herein;

[0083] Figure 3E A schematic view of an alternative object detection model according to one embodiment of the teachings herein is shown;

[0084] Figure 4A A flowchart of a general method according to an embodiment of the present invention is shown;

[0085] Figure 4B A flowchart of a general method according to an embodiment of the present invention is shown;

[0086] Figure 4C A flowchart of a general method according to an embodiment of the present invention is shown;

[0087] Figure 4D A flowchart of a general method according to an embodiment of the present invention is shown;

[0088] Figure 5A A component view of a software module device according to an embodiment of the teachings herein is shown;

[0089] Figure 5B A component view of a software module device according to an embodiment of the teachings herein is shown;

[0090] Figure 5C A component view of a software module device according to an embodiment of the teachings herein is shown;

[0091] Figure 5D A component view of a software module device according to an embodiment of the teachings herein is shown;

[0092] Figure 6A A component view of a device including circuit modules according to embodiments taught herein is shown;

[0093] Figure 6B A component view of a device including circuit modules according to embodiments taught herein is shown;

[0094] Figure 6C A component view of a device including circuit modules according to embodiments taught herein is shown;

[0095] Figure 6D A component view of a device including circuit modules according to embodiments of the teachings herein is shown; and

[0096] Figure 7 A schematic view of a computer-readable medium carrying computer instructions, which, when loaded into and executed by a controller of a device, enable the device to implement embodiments of the invention. Detailed Implementation

[0097] Figure 2A A schematic view of an object detection device 100 according to an embodiment of the present invention is shown. The object detection device 100 includes a controller 101, an image receiving device 112, and a memory 102. In one embodiment, the image receiving device 112 is an image sensor included in a camera module, or alternatively, a camera module. As those skilled in the art will understand, the object detection device 100 may include a controller 101, and the image receiving device 112 may include a controller; however, for the purposes of this teaching, they will be considered to be the same controller 101 in order to cover all possible variations in the location where image processing occurs precisely.

[0098] In one embodiment, image receiving device 112 is a communication interface for receiving image data therethrough. In one such example, image data is received from memory 102, where the image receiving device is a memory interface. In an alternative embodiment of this kind, image data is received from a remote source, where the image receiving device is a communication interface, such as that discussed below with reference to 103.

[0099] The controller 101 is configured to receive image data representing one or more images (or at least a portion of images) from the image receiving device 112, and to perform object detection on the received image(s)(s)(s). The image receiving device 112 may be included in the object detection device 100 by being housed in the same housing as the object detection device 100 or by being connected to it via a wired or wireless connection.

[0100] It should be noted that the object inspection device 100 may include a single device or may be distributed across several devices and apparatuses.

[0101] Controller 101 is also configured to control the overall operation of object detection device 100. In one embodiment, controller 101 is a graphics controller. In one embodiment, controller 101 is a neural processing controller. In one embodiment, controller 101 is a general-purpose controller. In one embodiment, controller 101 is a combination of a graphics controller, a neural processing controller, and / or a general-purpose controller. As those skilled in the art will understand, there are many alternatives regarding how to implement the controller, such as additionally or as alternatives using field-programmable gate arrays, ASICs, GPUs, NPUs, etc. For the purposes of this application, all these possibilities and alternatives will be simply referred to as controller 101.

[0102] It should also be noted that in one embodiment, some or all of the processing is performed remotely, wherein the local controller 101 is configured to provide input data to a remote processing unit, such as a cloud server, thereby causing the remote processing unit to perform the processing and receive the results of such processing as output from the remote processing unit. For the purposes of this application, such possibilities and alternatives will also be simply referred to as controller 101, whereby controller therefore represents both the local controller and the remote processing unit.

[0103] Memory 102 is configured to store data such as image data, settings, and computer-readable instructions that, when loaded into controller 101, indicate how the object detection device 100 will be controlled. Memory 102 may include several memory cells or devices, but they will be considered as part of the same overall memory 102. There may be a memory cell for a display device storing graphic data, a memory cell for an image receiving device storing settings, a memory for a communication interface (see below) storing settings, and so on. As those skilled in the art will understand, there are many possibilities for how to choose where the data should be stored, and thus the general-purpose memory 102 for the object detection device 100 includes any and all such memory cells for the purposes of this application. As those skilled in the art will understand, there are many alternatives for how the memory can be implemented, such as using non-volatile memory circuitry, such as EEPROM memory circuitry, or using volatile memory circuitry, such as RAM memory circuitry. For the purposes of this application, all these alternatives will simply be referred to as memory 102.

[0104] It should be noted that the teachings of this paper reveal the use of object detection device 100 in many areas of digital imaging, including digital photography (e.g., in smartphones), augmented vision, computer vision, mixed or augmented reality systems, image retrieval, industrial applications, automation, robot vision, autonomous navigation or driving, and video surveillance, where, for example, Figure 2A The basic object detection device 100 in the middle.

[0105] It should be noted that any objects detected in an image do not need to be displayed to the user, nor does the image itself need to be displayed. In some embodiments, object detection will play a role in control applications such as surveillance, monitoring, or automation applications, where visual output of images or objects is not required. This also applies to... Figure 2B and 2C The embodiments discussed are even those focused on viewing devices.

[0106] Figure 2B A schematic view of an object detection device as a viewing device 100 according to an embodiment of the present invention is shown. In this embodiment, the viewing device 100 is a smartphone or tablet computer. In such embodiments, the viewing device further includes a display device 110, which may be a touchscreen, and an image receiving device 112 may be a series of cameras of a smartphone or tablet computer. In such embodiments, a controller 101 is configured to receive images from the series of cameras 112, process the images, and most likely (but not necessarily) display the images on the display device 110.

[0107] Figure 2C A schematic view of an object detection device as an optical perspective (OST) (including video perspective) viewing device 100 according to an embodiment of the present invention is shown. The viewing device 100 is a perspective device in which a user views through one end and sees a real object in the line of sight at the other end of the viewing device 100.

[0108] In one embodiment, the viewing device 100 is a head-mounted viewing device 100 to be worn by a user. Figure 2C (Not explicitly shown) for observation through viewing device 100. In one such embodiment, viewing device 100 is arranged as eyeglasses or other eye-wearing devices including goggles to be worn by a user.

[0109] In one embodiment, the viewing device 100 is configured to be handheld, so that a user can lift the viewing device 100 to observe through it.

[0110] In one embodiment, the viewing device 100 is arranged to be mounted, for example, on a tripod, so that a user can install the viewing device 100 in a convenient device for observation. In one such embodiment, the viewing device 100 may be mounted on the dashboard of a car or other vehicle.

[0111] In one embodiment, the viewing device 100 is a digital imaging device for providing enhanced vision (e.g., night vision).

[0112] In one embodiment, the viewing device is a virtual, augmented, or mixed reality device for providing virtual reality, augmented reality, or mixed reality to a user.

[0113] The viewing device 100 includes an image receiving device 112 for receiving images and a display device 110 for presenting the captured and processed images to a viewer. (Refer to the above...) Figure 2A As disclosed, the image receiving device 112 may be remote and is included in the object detection device 100 via a connection to the object detection device 100.

[0114] Next, we will... Figure 2A , 2B Simultaneous reference is made with the 2C object detection device 100.

[0115] In one embodiment, the object detection device 100 may further include a communication interface 103. The communication interface 103 may be wired and / or wireless. The communication interface may include several interfaces.

[0116] In one embodiment, the communication interface 103 includes a USB (Universal Serial Bus) interface. In one embodiment, the communication interface 103 includes an HDMI (High-Definition Multimedia Interface) interface. In one embodiment, the communication interface 103 includes a DisplayPort interface. In one embodiment, the communication interface 103 includes an Ethernet interface. In one embodiment, the communication interface 103 includes a MIPI (Mobile Industry Processor Interface) interface. In one embodiment, the communication interface includes an analog interface, a CAN (Controller Area Network) bus interface, an I2C (Integrated Circuit Interoperability) interface, or other interfaces.

[0117] In one embodiment, communication interface 103 includes a radio frequency (RF) communication interface. In one such embodiment, communication interface 103 includes Bluetooth. TM Interface, WiFi TM Interface, ZigBee TM Interface, RFID TM (Radio Frequency Identifier) ​​interface, Wireless Display (WiDi) interface, Miracast interface, and / or other RF interfaces typically used for short-range RF communication. In an alternative or supplementary embodiment of this type, communication interface 103 includes a cellular communication interface, such as a fifth-generation (5G) cellular communication interface, an LTE (Long Term Evolution) interface, a GSM (Global System for Mobile Communications) interface, and / or other interfaces typically used for cellular communication. In one embodiment, communication interface 103 is configured to communicate using the UPnP (Universal Plug and Play) protocol. In another embodiment, communication interface 103 is configured to communicate using the DLNA (Digital Living Network Device) protocol.

[0118] In one embodiment, the communication interface 103 is configured to enable communication via more than one of the example techniques given above. As examples, a wired interface such as MIPI can be used to establish an interface between a display device, a controller, and a user interface, and WiFi, for example... TM The wireless interface can be used to enable communication between the object detection device 100 and an external host device (not shown).

[0119] The communication interface 103 can be configured to enable the object detection device 100 to communicate with other devices, such as other object detection devices 100 and / or smartphones, internet tablets, computer tablets or other computers, media devices such as televisions, game consoles, video viewers or projectors (not shown), or image receiving devices for receiving image data.

[0120] User interface 104 can be included in object detection device 100 (only in...). Figure 2B (as shown in the diagram). Alternatively or additionally, user interface 104 (at least a portion thereof) may be remotely included in object detection device 100 via communication interface 103. User interface 104 is therefore not a physical component of object detection device 100, but is implemented by receiving user input via communication interface 103 through a remote device (not shown). An example of such a remote device is a game controller, mobile phone, tablet computer, or computer.

[0121] This article and such as about Figure 2A , 2B The object detection device 100 disclosed in 2C is configured to perform object detection using multi-scale CNNs (e.g., ResNet SSD, Feature Pyramid Network (FPN), MS-CNN, SSD-MSN, etc.).

[0122] Figure 3A A schematic view of a general object detection model 300A according to an embodiment of the teachings herein is shown. The object detection model 300A is arranged to be executed by a controller 101 on an object detection device 100 according to the present invention, a software module device 500 according to the present invention (see FIG. 5), and a device 600 including a circuit module according to the present invention, which is being utilized by the method herein or according to the present invention (see FIG. 6). It should be noted that different parts of the object detection model may be executed by different parts of the corresponding execution devices.

[0123] The object detection model 300A includes an image data receiver 312 that receives image data corresponding to an image, as discussed above. For the purposes of this teaching, it will be assumed that the image data includes indications of one or more objects to be detected (i.e., classified). Therefore, the image data has already undergone processing such as segmentation. In prior art systems, the image data would simply undergo object classification (i.e., object detection) to provide object classification.

[0124] Object detection typically works by running data associated with a hypothetical object through a multi-scale CNN, which produces a confidence value C indicating the confidence that a hypothetical object belongs to a particular class among all hypothetical objects. As is known, applications employing object detection use a confidence threshold because many potential objects exist in an image. If the confidence falls above the class threshold, the object is considered to belong to that class. However, if the threshold is set too low, many false positives may occur, and therefore a high threshold is desired to avoid such false positives. As discussed in the above summary, the inventors have recognized that a problem with multi-scale CNNs is that the location of some objects will affect (reduce) the confidence of the detection, resulting in many objects not receiving a confidence score above the threshold and therefore not being detected correctly.

[0125] The inventors propose to address the problems recognized by the inventors and discussed herein by taking into account the position of the object during the detection process.

[0126] Therefore, the teachings of this paper are aimed at object detection devices that utilize multi-scale CNNs (such as ResNet SSD) for object detection (such as those related to...). Figure 2A , 2B (And those devices disclosed by 2C).

[0127] like Figure 1A As shown, object 11 is located at a distance d in the image. Distance d can be expressed as a vertical distance VD or a horizontal distance HD, or both. As mentioned above, the exact measurement of distance is not critical, as distance can be measured in many ways and in many directions. However, a typical way to measure distance is in pixels from the origin of the image coordinate system. In one embodiment, the distance to the starting point of object 11 in that direction is determined (e.g., ...). Figure 1A (In one embodiment, the distance to the center of object 11 is determined; in another embodiment, the distances to the start and end points of object 11 are determined, including the extension of object 11.)

[0128] The inventor proposed, for example Figure 3AThe object detection model shown performs a risk classification 314 for the confidence level of erroneous representations. If such risk exists, or more precisely if the risk is considered significant, the object detection is compensated accordingly 315 so that objects 320 can be detected correctly.

[0129] In one embodiment, risk is classified by determining whether object 11 is a horizontal or vertical object. As described above, object 11 is classified as a vertical object if its height H is greater than its width, or by a constant multiplied by its width W; that is, if H > cW, the object is vertical, where c is a constant and where c is 1, 1.1, 1.2 (to name just a few examples). Similarly, object 11 is classified as a horizontal object if its width W is greater than its height H, or by a constant multiplied by its width; that is, if W > cH, the object is vertical, where c is a constant and where c is 1, 1.1, 1.2 (to name just a few examples). When the constant C is 1, these two tests are mutually exclusive, and only one needs to be performed.

[0130] In one embodiment, risk is classified by determining the distance d from object 11 to image 10. In one embodiment, the distance d to the center of object 11 is determined. In an alternative or additional embodiment, the distance d to object 11 is determined, and the distance d includes the extent of object 11.

[0131] If the image is located at a distance corresponding to a multiple of the confidence distance with lower confidence (or within the margin distance of that distance) (as referenced) Figure 1B If the distance d (discussed in the invention summary) is considered to be at risk, then the object is considered to be in a risky location. In one embodiment, the margin distance is 0, where the object is considered to be in a risky location only if the distance d is a multiple of the confidence distance. In one embodiment, the margin distance is 1, 2, or 5 pixels. In one embodiment, the margin distance depends on the confidence distance cd, and is, for example, 5%, 10%, 20%, or 25% of the confidence distance. In one embodiment, the margin distance depends on the resolution of the image, and is, for example, 1%, 2%, or 5% of the resolution. In one embodiment, the margin distance depends on the size of the object in the relevant direction, and is, for example, 1%, 2%, 5%, or 10% of the size of the object in the relevant direction.

[0132] In one embodiment, the confidence distance is not a single distance (such as a pixel), but rather a range of pixels indicating a range or region with lower confidence.

[0133] In one embodiment, points, ranges, or regions with low confidence do not begin at multiples of the confidence distance from the image edge, but a distance constant is considered when comparing distance d to the confidence distance. In one embodiment, the distance constant is added to the multiplied confidence distance, but those skilled in the art will understand that it does not matter whether the distance constant is added to the distance or to the multiplied confidence distance. The distance constant is equal to or greater than 0 and less than the confidence distance (within the corresponding extension).

[0134] In one such embodiment, when determining that an object is at risk based on distance, it is determined whether the object is at risk because it is located at a horizontal distance matching the confidence distance (i.e., located at or within the range of a distance constant and possibly considering the distance constant) and / or located at a vertical distance matching the confidence distance. If the object is at risk due to any one or both of these distances matching the confidence distance, compensation detection is performed regardless of the object's extent. In one such embodiment, if the matching distance corresponds to the extent of object 11, the compensation can be more effective (i.e., applied to a higher degree).

[0135] In one embodiment, an object is determined to be horizontal or vertical only if it is determined to be in a risky location. In such embodiments, the location is refined by determining whether it is at a corresponding distance from a risky location (vertical distance for a vertical object and horizontal distance for a horizontal object). If not, there is no risk. This has the benefit of classifying objects only when necessary.

[0136] In one embodiment, before determining whether an object is in a risky location, it is determined whether the object is horizontal or vertical, wherein when determining whether an object is in a risky location, only the corresponding distances (vertical distance for a vertical object and horizontal distance for a horizontal object) are used. This has the advantage of studying only the corresponding distances rather than both.

[0137] In one embodiment, whether an object is at risk is determined based on both horizontal and vertical distances. The distance d is thus both the horizontal and vertical distances, each compared to a multiple of the corresponding confidence distance (possibly considering a distance constant).

[0138] Although the teachings in this paper focus on adaptation in only one direction, they can be used to adapt in two directions. The adaptations can be performed simultaneously or sequentially.

[0139] More robust testing was achieved by adapting to both horizontal and vertical extensibility.

[0140] As discussed in the invention description, the size of the object is not the primary factor; rather, its position is. However, the inventors have recognized that if the object is small compared to the resolution—that is, if the object has an extension below a size threshold in the relevant direction (vertical for vertical objects and horizontal for horizontal objects)—the confidence distance will be reduced by a factor of two or halved from the higher confidence distance representing larger objects. Alternatively, if the object has an extension above a size threshold in the relevant direction (vertical for vertical objects and horizontal for horizontal objects), the confidence distance will be increased by a factor of two or halved from the lower confidence distance representing smaller objects. Alternatively, to put it slightly differently, if an object has an extension in the relevant direction (vertical for vertical objects and horizontal for horizontal objects) that falls above the size threshold, then the object is considered to be in a risky location if the distance d is determined to be an even multiple of the confidence distance (or within its margin), and if an object has an extension in the relevant direction (vertical for vertical objects and horizontal for horizontal objects) that falls below the size threshold, then the object is considered to be in a risky location if the distance d is determined to be an arbitrary multiple of the confidence distance (or within its margin), with the confidence distance thus specified for smaller objects.

[0141] In one embodiment, the size threshold is 64, 128, or 256 pixels. In another embodiment, the size threshold depends on the resolution in the relevant direction, for example, 1 / 4, 1 / 8, or 1 / 16 of the resolution.

[0142] As can be seen, risk classification 314 can be performed in many different ways and in different orders. As can be seen from the above embodiments, risk classification 314 can be performed before or after object detection 315, because the confidence level C has been generated and will be compared with the class threshold.

[0143] In one embodiment, risk classification 314 is performed after object detection 315, and the provided confidence level C is taken into account.

[0144] In one such embodiment, risk classification 314 determines whether the provided confidence level C fails to reach the (relevant) class threshold, i.e., the confidence level required to classify the object as belonging to a particular class. If the provided confidence level C fails to reach the (relevant) class threshold, risk classification 314 determines whether to compensate object detection 315 accordingly by performing any of the risk classifications described above, i.e., determining the object's location and / or class as described above.

[0145] In an alternative or additional embodiment of this kind, risk classification 314 determines whether the provided confidence level C is within the confidence level range indicating risk, and if so, risk classification 314 determines whether object detection 315 should be compensated accordingly by performing any of the risk classifications described above (i.e., determining the location and / or class of the object as above). In one embodiment of this kind, the confidence level range is a range below a class threshold, which makes this embodiment directly the embodiment described above. In an alternative embodiment, the confidence level range is based on a class threshold, such as 75-100% of the class threshold, 50-100% of the class threshold, or 25-100% of the class threshold. In an alternative embodiment, the range is based on an absolute number, such as 0.75-1, 0.5-1, or 0.25-1.

[0146] In an alternative or additional embodiment of this kind, the confidence interval depends on the location, and more specifically, on the distance of the correlation (i.e., closest) multiple of the relative confidence distance. Shorter distances provide lower confidence intervals (i.e., achieving lower confidence levels).

[0147] The benefit of performing risk classification 314 after a confidence level C has already been provided is that the detection is compensated only when it is actually needed (as indicated by the low confidence level value).

[0148] If an object has been identified as being at risk (whether before or after the (initial) object detection has been performed), then the object detection is compensated accordingly via compensated detection 315.

[0149] Figure 3B A schematic view of an alternative object detection model 300B according to one embodiment of the teachings herein is shown. The object detection model 300B is arranged to be executed by a controller 101 of an object detection device 100 according to the present invention, a software module device 500 according to the present invention (see FIG. 5), and a device 600 including a circuit module according to the present invention or according to the present invention that is being utilized. It should be noted that different parts of the object detection model may be executed by different parts of the corresponding execution devices.

[0150] Object detection model 300B is Figure 3A An alternative embodiment of the object detection model 300A is provided, and an adapter 316 is also included. Figure 3B In the example, adapter 316 is included in compensated detection 315. Regarding Figure 3A The embodiments discussed involve Figure 3B Examples, and as discussed with respect to those embodiments, can be performed regarding Figure 3B The discussed embodiments enable risk classification 314 to be performed before or during compensated detection (i.e., when a confidence value has already been detected).

[0151] In one embodiment, the compensated detection 315 includes adapting the detection 316 by means of parameters of the adapted object detection, such as a threshold or confidence level provided for the detection.

[0152] In one such embodiment, compensated detection 315 includes adaptation 316 detection by lowering a class threshold of the guessed or associated class. In one such embodiment, the threshold is lowered by a certain amount based on a multiple of the confidence distance cd, and the closer the object is to a multiple of the confidence distance, the lower the threshold. As those skilled in the art will understand, lowering the threshold has substantially the same effect as increasing the confidence, and thus the two are alternatives to each other and are substantially interchangeable.

[0153] Alternatively, instead of lowering the threshold, the confidence level C for the object 11 to be determined or confirmed can be increased as adaptation 316. In one such embodiment, the confidence level is increased by a certain amount based on a multiple of the confidence level distance cd, with the confidence level increasing the closer the object is to a multiple of the confidence level distance.

[0154] In one embodiment, these two methods are combined, where the threshold is simultaneously decreased as the confidence level increases. This has the benefit of avoiding either excessively decreasing the threshold or excessively increasing the confidence level, thus preventing another type of object from being incorrectly classified simply because the threshold has been decreased too much.

[0155] In short, Figure 3B The main principle of Model 300B is that it is based on additional steps in convolutional neural networks with multi-scale structures, such as those discussed in the background section and the summary of the invention, where the confidence level of candidate objects is adjusted based on an estimate of the influence of the precise location of objects in an image. This estimate depends on location, object size and aspect ratio, and other underlying factors relevant to the design of the multi-scale network model. An alternative, but very similar, approach is to dynamically adjust the thresholds of individual candidate objects rather than their confidence levels, since positive identification is determined by whether the confidence level is above a certain threshold.

[0156] In another embodiment, adaptation 316 and / or risk classification 314 are mapped to (or remotely) tables stored in memory 102, and the controller is configured to perform a lookup for each object based on object attributes or functions thereof. Such a table structure will be multi-dimensional, so the values ​​used for adaptation (threshold and / or confidence adaptation) can depend on the object's size, alignment, aspect ratio, and also, for example, which class of the object (e.g., person, bottle, horse, etc.). Therefore, the table lookup will provide an adaptation 316 for compensated detection 315 based on an implicit classification 314 represented by values ​​stored in the table. In one such embodiment, the controller is configured to classify 314 whether the object 11 to be detected is at risk of being incorrectly detected based on distance d, and adapt 316 object detection parameters by retrieving the parameters to be used via a table lookup, where the lookup table carries the parameters to be used for the object according to distance d.

[0157] In another embodiment, a simpler neural network (or other machine learning algorithm or model) corresponding to model 300B is trained to determine the fit, and during the execution of the CNN for object detection, the smaller neural network 300B is used to calculate the risk and / or fit.

[0158] In some embodiments, alignment compensation functionality can be discovered, for example, by “scanning” the input of an object detector with objects of known size, position, and aspect ratio (“golden truth”), and comparing the output of the object detector with the golden truth. A regressor can be trained using the object detector output as input and with the aspect ratio, size, and / or position based on the golden truth as the target output.

[0159] Figure 3C A schematic view of an alternative object detection model 300C according to one embodiment of the teachings herein is shown. The object detection model 300C is arranged to be executed by a controller 101 of an object detection device 100 according to the present invention, a software module device 500 according to the present invention (see FIG. 5), and a device 600 including circuit modules according to the present invention or according to the present invention, which is being utilized by the method herein. It should be noted that different parts of the object detection model may be executed by different parts of the corresponding execution devices.

[0160] Figure 3D It shows how to follow the teachings of this article according to... Figure 3C The model 300C compares schematic views of two images. Simultaneously referencing... Figure 3C and 3D To describe Figure 3C Model 300C.

[0161] Object detection model 300C is Figure 3AAn alternative embodiment of the object detection model 300A is provided, and the compensated detection 315 includes a shifting module 317 for shifting the image 10. In such embodiments, the image 10 can be shifted in one or both directions (vertical and / or horizontal). Since the detection is affected by the position of the object 11, any shift in the image will effectively place the object in a different position and thus provide different detection results. Figure 3D The diagram illustrates how an image is shifted 317, wherein object 11 is located at a first distance d1 in a first instance of image 10-1 and at a second distance d2 in a second instance of image 10-2. Figure 3D In this context, displacement affects the horizontal distance laterally; however, it should be noted that displacement can occur in any one or both of the horizontal and vertical directions. It should also be noted that displacement can occur in any horizontal or vertical direction, i.e., left or right and / or up or down, even in... Figure 3D The middle is exemplified as to the left.

[0162] exist Figure 3D In this context, object 11 has been shifted by a distance equal to the difference between the first distance D1 and the second distance D2, i.e., the shift distance sd = d2 - d1. In one embodiment, the object is shifted relative to the image. In an alternative embodiment, the object is shifted along with the image; that is, all or part of the image is translated along with the object. The shift is performed in a general direction adapted to give a shift in the horizontal or vertical direction, or a shift in both directions.

[0163] To ensure that the shift does not simply place the object 11 in a different risky position, the shift distance is different from the confidence distance. In one embodiment, the shift distance is less than the confidence distance, sd < cd. Since the confidence drops rapidly at or near the risky position, the shift distance does not need to be large, but in one embodiment, it is greater than the margin distance (md) to ensure that the object is shifted away from the risky position, sd > md. In one such embodiment, when the shift direction is in the direction towards the risky position, the shift distance is greater than twice the margin distance to ensure that the object is shifted away from the risky position, i.e., sd > 2×md. Thus, in one embodiment, the shift distance is in the range md < sd < cd. In one embodiment, the shift distance depends on the resolution of the image and is, for example, 1%, 2%, or 5% of the resolution. In one embodiment, the shift distance depends on the confidence distance cd and is, for example, 5%, 10%, 25%, or 50% of the confidence distance. In one embodiment, the shift distance depends on the size of the object 10, particularly the size in the relevant direction H or W, and is, for example, 5%, 10%, 25%, or 50% of the size of the object 11. In one embodiment, the shift distance is the distance from the object 10 to the position between the two risky positions, i.e., the shift distance is equal to half the confidence distance plus the distance from the object to a relevant multiple of the confidence distance). In one embodiment, the shift distance is determined based on one object 10. However, in some cases, there are several objects in the risky position. To overcome this problem, one embodiment provides a shift and a shift distance for the image of one or more of the several objects for target object detection such that all objects can be correctly detected after the corresponding shift. In an alternative embodiment, the shift distance is determined such that all (or at least most) of the objects to be detected are moved away from the risky position. In such embodiments, the shift distance is the average of the individual shift distances determined for each object as if it were the only object to be detected. In one embodiment, the controller is further configured to determine the shift distance by also ensuring that no (or at least few) objects that were not previously in the risky position are moved to the risky position. It should be noted that since the shift provides two detections to be compared, there is a minimal risk that the object will not be successfully detected in one of the detections (at least due to the effect of the distance into the image).

[0164] The ways for determining the shift distance are not exclusive but can be combined.

[0165] In one embodiment, if risk classification 314 indicates that object 11 is at risk, then image 10 is shifted. This avoids any unnecessary shifting of the image. However, in one embodiment, the image can be shifted regardless of the risk classification, whereby the risk classification is considered part of the shift because all images are considered to be at risk and are compensated for. Therefore, in such embodiments, the specific and separate execution of risk classifications is optional, as they are implicit in the structure of the model. Thus, image shift 317 is considered to be risk classification 314.

[0166] In one embodiment, the same image 10 is shifted, such that two instances of the same image are processed and detected. Figure 3C This is indicated by feedback returning from detector 318 to detector 318 via shift 317. Figure 3D In this context, the first instance 10-1 is therefore an unshifted instance of image 10, while the second instance 10-2 is a shifted instance of image 10. This allows the object detection device 100 to provide two confidence levels, C1 and C2, one for each instance, and to determine the object detection result based on the confidence level or detection outcome (in...). Figure 3D The comparison of confidence levels C1 and C2 (represented by 319) determines whether either of them provides a successful detection.

[0167] In an image containing multiple objects, some candidate objects may be at risk, while others are not. In such cases, shifting might introduce new objects into the risk. The amount of compensation to be given to the objects will be a trade-off based on several factors. One practical solution would be to shift by an amount that statistically maximizes the probability (perhaps half the current value) that at least one of the two shifts will provide a risk-free object. Alternatively, if, for example, there are two objects at risk and they are aligned in a similar manner, the shift can be based on what is most meaningful for those two cases.

[0168] This allows the object detection device 100 to have a second chance to correctly or successfully detect the object by shifting the image and feeding it back after the first detection is unreliable or unsuccessful, and then comparing the detection results. If a poor detection result is a consequence of the object being in a risky location, the shift will handle the situation (where comparison 319 acts as a risk classifier 314).

[0169] However, if the poor detection is due to another reason, shifting will not solve the problem. To prevent unnecessary processing of shifted images, risk classification 314 is advantageously performed before shifting the images.

[0170] In one embodiment, the image to be shifted is not the same as the first image, but a subsequent image, such as the next frame in a video stream. In such embodiments, the second frame (second image) is shifted 317 if the first frame (first image) is determined to be in a risky position and / or provides low confidence.

[0171] When the detection is compensated by shift, the results of 319 can be compared, and the detection result with the highest confidence can be selected as the detected object 320 (assuming the detection result corresponds to a sufficiently high confidence level).

[0172] The shifting of image 317 is one way of compensating detection 315 of an object at risk, and any of the ways of determining whether an object should receive compensating detection (i.e., risk classification 314 taught herein) also applies to such embodiments.

[0173] In one embodiment, the shift distance is a function of the object's movement and can be determined based on the object's position and / or aspect ratio. In one such embodiment, risk classification 314 is arranged to predict whether the object will be located at a risk position in a subsequent or next image, and if so, to preemptively shift the image 317 as part of compensated detection 315 before any detection is performed. This prediction can also predict the distance the object will need to be shifted, i.e., determine the shift distance to ensure that the object is not located at a risk position in a subsequent or second image 10-2.

[0174] Figure 3E A schematic view of an alternative object detection model 300D according to one embodiment of the teachings herein is shown. The object detection model 300D is arranged to be executed by a controller 101 of an object detection device 100 according to the present invention, a software module device 500 according to the present invention (see FIG. 5), and a device 600 including a circuit module according to the present invention, which is being utilized according to the present invention or according to the present invention (see FIG. 6). It should be noted that different parts of the object detection model can be executed by different parts of the corresponding execution devices.

[0175] Object detection model 300D is Figure 3A Alternative embodiments of the object detection model 300A, and also include Figure 3B Model 300B is compatible with 316 and Figure 3D The shift of model 300C.

[0176] This allows for compensation detection by selecting parameters for the 316 adaptation, shifting the image, and / or performing both. Due to the low overhead of the 316 adaptation, it is advantageously used to reduce the required computational workload. However, since shifting provides a comparative indicator of how well the compensation has been performed, it is advantageously used to ensure correct or successful detection. Combining these two allows the target device to determine whether the parameter adaptation was successful, adjust the parameters if unsuccessful, and continue using only the 316 adaptation if successful.

[0177] In such embodiments, controller 101 is configured to initially (or for one or more images) utilize both adaptation 316 and shift 317 to set parameters for adaptation 316, thereby achieving appropriate results (appropriate results mean a sufficiently high detection success rate required by the designer or application). In one embodiment, the combination method is initially performed to initiate the system. In one embodiment, the combination method is performed when a new type of object is to be detected. In one embodiment, the combination method is performed when the detection success rate is below a threshold success rate. In one embodiment, the combination method is executed as a background process to verify compensated detection 315.

[0178] The detection parameters are adapted so that future detections will provide smaller differences in the object detection results. In one embodiment, the confidence is increased and / or the threshold is decreased if the second object detection (i.e., the object detection of the shifted image) provides a more reliable result (i.e., with higher confidence). In another embodiment, the confidence is increased and / or the threshold is decreased based on the difference in confidence provided by the first object detection (i.e., the unshifted image) and the second object detection (i.e., the shifted image). In one such embodiment, the confidence is increased and / or the threshold is decreased by an amount equal to the difference in confidence multiplied by a constant K, where the constant K is 0.25, 0.5, or 1, which allows the adaptation to make any difference in confidence negligible (at least for repeated adaptations and shifts of a series of images, such as when initializing or training object detection).

[0179] In one embodiment, a combined approach is used such that the adaptation 316 is performed locally by the controller 101 of the object detection device 100, and the shift and subsequent comparison 319 are performed by the remote controller. This enables the local controller to manage the computational load even when it has limited capacity, or when it does not use much of its capacity and leaves the larger computations to the remote controller, which may have more resources.

[0180] Therefore, in one embodiment, the combination method is arranged to utilize shifting to establish an adaptation.

[0181] Therefore, in one embodiment, the combination method is also or alternatively arranged to utilize shifting to align or correct the fit.

[0182] Therefore, in one embodiment, the combined method is also or alternatively arranged to utilize shifting to verify the fit.

[0183] It should be noted that even though the description herein focuses on one relevant direction, two directions can be considered relevant, and risk classification is performed simultaneously for both directions. Risk classification can stipulate that if an object is in a corresponding risk location (as described above) based on its location, it is considered to be under an incorrectly classified risk. Therefore, the above embodiments can be combined (either individually or in combination) with respect to the processing of two directions.

[0184] As those skilled in the art will understand, models 300A, 300B, 300C, and 300D are examples of models that can be included in a general CNN model, and will therefore be considered as additions or supplements to such CNN models.

[0185] As those skilled in the art will also understand, even though the description herein focuses primarily on a single image, the teachings herein can be advantageously applied to a series of images, such as a video stream. In particular, regarding... Figure 3C and 3D The embodiments discussed can be applied to this type of image series.

[0186] Figure 4A A general flowchart of the method taught in this paper is shown. This method corresponds to... Figure 3A Object detection model 300. Image data 410 is received from image receiving devices 112, 312, wherein the image data has been or will be processed in a CNN for object detection in the image data 420. It is determined whether the object to be detected is at risk 430, and if so, detection is adapted accordingly 440, and the object is detected 420 and provided 450.

[0187] It should be noted that if an object is at risk, the actual object detection can be performed before or after classification, or both before and after classification.

[0188] Figure 4B A general flowchart of the method taught in this paper is shown. This method corresponds to... Figure 3B Object detection model 300. This flowchart represents the process of... Figure 4A The flowchart in the diagram represents the implementation of the method, and similarities will not be discussed but simply assumed to be included. Figure 4B In the covered embodiments described above, adaptation 440 is performed by compensating for detection 442. Compensation can be performed by adapting the confidence level, adapting the threshold used, or adapting both.

[0189] Figure 4C A general flowchart of the method taught in this paper is shown. This method corresponds to... Figure 3C Object detection model 300. This flowchart represents the process of... Figure 4A The flowchart in the diagram represents the implementation of the method, and similarities will not be discussed but simply assumed to be included. Figure 4C In the covered and described embodiments, adaptation 440 is performed by shifting 444 objects and / or images, processing 446 the shifted images / objects, and selecting 448 the best result.

[0190] Figure 4D A general flowchart of the method taught in this paper is shown. This method corresponds to... Figure 3E Object detection model 300. This flowchart illustrates the process... Figure 4A The implementation of the method represented by the flowchart in the diagram, where Figure 4B and Figure 4C The implementations have been combined, and similarities will not be discussed but simply assumed to be included. Figure 4D In the covered and as described above embodiments, by means of Figure 4B (ALT1) or according to Figure 4C (ALT2) Select the 435 adapter to adapt to the 440. It should be noted that the selection does not need to be exclusive, but both alternatives can be selected.

[0191] Figure 5A A component view of a software module (or component) device 500 according to an embodiment of the teachings herein is shown. The software module device 500 is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein and for enabling the object detection device 100 to perform according to... Figure 4A The method.

[0192] The software module device 500 includes a software module (510) for receiving image data. The software module device 500 also includes a software module 520 for processing the image data using a CNN to detect objects in the image data, a software module 530 for determining whether an object to be detected is at risk, and a software module 540 for adapting the detection accordingly if so. The software module device 500 also includes a software module 550 for providing the object.

[0193] Figure 5B A component view of a software module (or component) device 500 according to an embodiment of the teachings herein is shown. The software module device 500 is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein and for enabling the object detection device 100 to perform according to... Figure 4B The method.

[0194] Software module device 500 representative Figure 5A The implementation of the software module device 500 in the document will be discussed, and similarities will not be discussed, but simply assumed to be included. Figure 5B In the covered embodiments described above, the software module 540 for adaptation includes a software module 542 for compensating for detection. The compensation can be performed by adapting the confidence level, adapting the threshold used, or adapting both.

[0195] Figure 5C A component view of a software module (or component) device 500 according to an embodiment of the teachings herein is shown. The software module device 500 is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein and for enabling the object detection device 100 to perform according to... Figure 4C The method.

[0196] Software module device 500 representative Figure 5A The implementation of the software module device 500 in the document will be discussed, and similarities will not be discussed, but simply assumed to be included. Figure 5C In the covered and described embodiments, the software module 540 for adaptation includes a software module 544 for shifting images, a software module 546 for processing shifted images, and a software module 548 for selecting the best result.

[0197] Figure 5D A component view of a software module (or component) device 500 according to an embodiment of the teachings herein is shown. The software module device 500 is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein and for enabling the object detection device 100 to perform according to... Figure 4D The method.

[0198] Software module device 500 representative Figure 5A The implementation of the software module device 500 in the document is... Figure 5B and 5C The combination of software module devices 500, and similarities will not be discussed, but are simply considered to be included. Figure 5D In the covered embodiments described above, the software module device 500 includes options for selecting which adapter to use, the adapter according to FIG. B, and / or according to... Figure 5C The software module 535 is adapted for use. In alternative or additional embodiments, the software module device 500 further includes a function for performing operations related to... Figure 4A , 4BThe software module refers to any, some, or all of the embodiments of the methods discussed in 4C and 4D. In alternative or additional embodiments, the software module device 500 also includes software modules for performing operations as described above. Figure 2A , 2B 2C and Figure 3A , 3B Software modules of any, some, or all of the configurations of the controllers discussed in 3C and 3D, wherein such software modules are considered to be included in software module 530 for classification and software module 540 for compensated detection.

[0199] Figure 6A A component view of a device 600 including a circuit module for object detection according to an embodiment of the teachings herein is shown. The device 600 including the circuit module for object detection is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein, and for enabling the object detection device 100 to perform according to Figure 4A The method.

[0200] The apparatus 600, which includes a circuit module for object detection, includes circuitry (510) for receiving image data. The apparatus 600 also includes a circuit module 620 for processing the image data using a CNN to detect objects in the image data, a circuit module 630 for determining whether an object to be detected is at risk, and a circuit module 640 for adapting detection accordingly if so. The apparatus 600 also includes a circuit module 650 for providing the object.

[0201] Figure 6B A component view of a device 600 including a circuit module for object detection according to an embodiment of the teachings herein is shown. The device 600 including the circuit module for object detection is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein, and for enabling the object detection device 100 to perform according to Figure 4B The method.

[0202] Device 600, which includes a circuit module for object detection, is indicated. Figure 6A The implementation of device 600, which includes a circuit module for object detection, is included, and similarities will not be discussed but simply assumed to be included. Figure 6B In the covered embodiments described above, the circuit module 640 for adaptation includes a circuit module 642 for compensation detection. The compensation can be performed by adapting the confidence level, adapting the threshold used, or adapting both.

[0203] Figure 6CA component view of a device 600 including a circuit module for object detection according to an embodiment of the teachings herein is shown. The device 600 including the circuit module for object detection is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein, and for enabling the object detection device 100 to perform according to Figure 4C The method.

[0204] Device 600, which includes a circuit module for object detection, is indicated. Figure 6A The implementation of device 600, which includes a circuit module for object detection, is included, and similarities will not be discussed but simply assumed to be included. Figure 6C In the covered and described embodiments, the circuit module 640 for adaptation includes a circuit module 644 for shifting the image, a circuit module 646 for processing the shifted image, and a circuit module 648 for selecting the best result.

[0205] Figure 6D A component view of a device 600 including a circuit module for object detection according to an embodiment of the teachings herein is shown. The device 600 including the circuit module for object detection is adapted for use in an object detection device 100 as taught herein, for providing object detection as taught herein, and for enabling the object detection device 100 to perform according to Figure 4D The method.

[0206] Device 600, which includes a circuit module for object detection, is indicated. Figure 6A The implementation of device 600, which includes a circuit module for object detection, is Figure 6B and 6C The combination of device 600, which includes a circuit module for object detection, is included, and similarities will not be discussed but are simply considered to be included. Figure 6D In the covered embodiments described above, the device 600, which includes a circuit module for object detection, includes options for selecting which adapter to use, the adapter according to FIG. B, and / or according to... Figure 6C The compatible circuit module 635.

[0207] In alternative or additional embodiments, the device 600, which includes a circuit module for object detection, further includes a function for performing... Figure 4A , 4B The circuit modules in any, some, or all of the method embodiments discussed in 4C and 4D. In alternative or additional embodiments, the device 600, which includes a circuit module for object detection, also includes circuit modules for performing actions such as those discussed in the 4C and 4D method embodiments. Figure 2A , 2B 2C and Figure 3A , 3BThe circuit modules of any, some, or all of the configurations of the controllers discussed in 3C and 3D, wherein such circuits are seen to be included in circuit module 630 for classification and circuit module 640 for compensated detection.

[0208] Figure 7 A schematic view of a computer-readable medium 120 carrying computer instructions 121, which, when loaded into and executed by the controller of the object detection device 100, enable the object detection device 100 to implement the present invention.

[0209] The computer-readable medium 120 can be tangible, such as a hard disk drive or flash memory, like a USB storage stick or cloud server. Alternatively, the computer-readable medium 120 can be intangible, such as a signal carrying computer instructions that enable the downloading of the computer instructions via a network connection such as an internet connection.

[0210] exist Figure 7 In the example, computer-readable medium 120 is shown as a computer disk 120 carrying computer-readable computer instructions 121, which is inserted into a computer disk reader 122. The computer disk reader 122 may be part of a cloud server 123 or other server, or the computer disk reader may be connected to a cloud server 123 or other server. The cloud server 123 may be part of the Internet or at least connected to the Internet. The cloud server 123 may alternatively be connected via a proprietary or dedicated connection. In one example embodiment, the computer instructions are stored on a remote server 123 and downloaded to the memory 102 of the object detection device 100 for execution by the controller 101.

[0211] The computer disk reader 122 may also be connected to (or possibly inserted into) the object detection device 100 for transmitting computer-readable computer instructions 121 to the controller of the object detection device 100 (assuming via the memory of the object detection device 100).

[0212] Figure 7 The illustration shows a scenario where object detection device 100 receives computer-readable computer instructions 121 via a server connection and a scenario where another object detection device 100 receives computer-readable computer instructions 121 via a wired interface. This allows the computer-readable computer instructions 121 to be downloaded into object detection device 100, thereby enabling object detection device 100 to operate and implement the invention disclosed herein.

Claims

1. An object detection device (100) comprising a controller (101) configured to detect objects using a multi-scale convolutional neural network, wherein, The controller (101) is also configured to: Receive (312, 410) image data representing an image (10), the image (10) including an object (11) to be detected, the object (11) being located at a distance to the edge of the image (10), wherein the distance is a horizontal distance and / or a vertical distance; By determining whether the distance is at a multiple of the confidence distance plus a distance constant, or by determining whether the distance is within the margin of the multiple of the confidence distance, the object (11) to be detected is classified (314, 430) based on the distance to determine whether it is at risk of being incorrectly detected; And if it is Compensate for (315, 440) object detection by adapting (316) object detection parameters.

2. The object detection device (100) according to claim 1, wherein, The controller is configured to adapt (316) the object detection parameters by lowering the class threshold.

3. The object detection device (100) according to claim 1, wherein, The controller is configured to adapt (316) the object detection parameters by increasing the confidence (C) for the object (11).

4. The object detection device (100) according to claim 2 or 3, wherein, The controller is configured to adapt (316) the object detection parameters by lowering the class threshold and by increasing the confidence (C) for the object (11).

5. The object detection device (100) according to claim 2 or 3, wherein, The controller is configured to reduce the class threshold by a certain amount based on the distance of the object (10) to a multiple of the confidence distance (cd).

6. The object detection device (100) according to claim 3, wherein, The controller is configured to increase the confidence (C) of the object (11) by a certain amount based on a distance that is a multiple of the confidence distance (cd).

7. The object detection device (100) according to any one of claims 1 to 3, wherein, The controller is configured to classify (314, 430) whether the object (11) to be detected is at risk of being incorrectly detected based on the distance during the compensated detection (315, 440).

8. The object detection device (100) according to claim 1, wherein, The controller is configured to classify (314, 430) whether the object (11) to be detected is at risk of being incorrectly detected based on the distance by retrieving parameters to be used via a lookup table, and to adapt (316) the object detection parameters, wherein the lookup table carries parameters to be used for the object according to the distance.

9. The object detection device (100) according to claim 8 further includes a memory (102) configured to store the lookup table.

10. The object detection device (100) according to claim 8 further includes a communication interface (103) for retrieving information from the lookup table.

11. The object detection device (100) according to any one of claims 1 to 3, wherein, The object detection device (100) is a smartphone or tablet computer.

12. The object detection device (100) according to any one of claims 1 to 3, wherein the object detection device (100) is an optical perspective device (100).

13. A method for object detection using a multi-scale CNN in an object detection device (100), wherein, The method includes: Receive (312, 410) image data representing an image (10), the image (10) including an object (11) to be detected, the object (11) being located at a distance to the edge of the image (10), wherein the distance is a horizontal distance and / or a vertical distance; By determining whether the distance is at a multiple of the confidence distance plus a distance constant, or by determining whether the distance is within the margin of the multiple of the confidence distance, the object (11) to be detected is classified (314, 430) based on the distance to determine whether it is at risk of being incorrectly detected; and if so... The object detection is compensated by adapting the object detection parameters (315, 440).

14. A computer-readable medium (120) carrying computer instructions (121) that, when loaded into and executed by a controller (101) of an object detection device (100), enable the object detection device (100) to perform the method according to claim 13.

15. A software module device (500) for object detection using a multi-scale CNN in an object detection apparatus (100), wherein the software module device (500) comprises: Software modules for receiving (312, 410, 510) image data representing an image (10), the image (10) including an object (11) to be detected, the object (11) being located at a distance to the edge of the image (10), wherein the distance is a horizontal distance and / or a vertical distance; A software module for classifying (314, 430, 530) whether the object (11) to be detected is at risk of being incorrectly detected based on the distance by determining whether the distance is at a multiple of the confidence distance plus a distance constant, or by determining whether the distance is within the margin of the multiple of the confidence distance. And if it is A software module for compensating for the object detection by adapting (316) the object detection parameters (315, 440, 540).

16. An object detection apparatus (600) including a circuit module for object detection using a multi-scale CNN, comprising: Circuit modules for receiving (312, 410, 610) image data representing an image (10), the image (10) including an object (11) to be detected, the object (11) being located at a distance to the edge of the image (10), wherein the distance is a horizontal distance and / or a vertical distance; By determining whether the distance is at a multiple of the confidence distance plus a distance constant, or by determining whether the distance is within the margin of the multiple of the confidence distance, the circuit module used to classify (314, 430, 630) whether the object (11) to be detected is at risk of being incorrectly detected is determined based on the distance; And if it is Circuit modules for compensating the object detection by adapting (316) object detection parameters (315, 440, 640).

Images