Systems, products, and methods for detecting and correcting sensing errors in sensing systems.

Abstract

The present disclosure relates to systems, products and methods for detecting and correcting perception errors in perception systems. A system for detecting and correcting perception errors in a perception system is disclosed. In operation, the system generates a list of detection targets based on perception data of a scene, which allows a list of background classes to be generated based on backgrounds in the perception data associated with the list of detection targets. For each detection target in the list of detection targets, the nearest background class is identified from the list of background classes. The vector can then be used to determine semantic features used to identify axioms. Optimal perception parameters are then generated, which are used to adjust perception parameters in an operating system to minimize perception errors.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-in-part of U.S. application No. 17 / 030,354 filed on September 23, 2020, which is a non-provisional patent application of U.S. provisional application No. 62 / 905,059 filed on September 24, 2019 and U.S. provisional application No. 62 / 984,713 filed on March 3, 2020, the entire contents of which are incorporated herein by reference.

[0003] This application also claims the benefit of and is a non-provisional patent application of US62 / 984,728 filed on March 3, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present invention relates to perception systems, and more particularly, to a perception system that uses semantic features to detect and correct perception errors. Background Art

[0005] Perception systems are commonly used for object recognition and tracking. In terms of current prospects, deep learning powers a major portion of state-of-the-art perception systems (see Reference No. 4 in the incorporated reference list). These systems are inherently difficult to interpret and understand, which makes reasoning about their successes and failures a difficult task. In addition, most perception systems today operate solely on appearance-based information, whether it is three-dimensional depth information from light detection and ranging systems (LiDAR) or visual information in the form of images from electro-optical sensors. When compared to human perception, state-of-the-art perception systems lack conceptual information about the physics of the world, ideas about relationships between entities, and task-specific context.

[0006] Additionally, today's perception systems suffer from issues related to false positive detections, missed detections, misclassifications, target track interruptions, category switching in track, etc. The reliability of perception systems is a major barrier to the adoption of autonomous systems. Therefore, to address these issues, current perception systems employ sensor and sensor modality redundancy to overcome the lack of perception performance. While redundancy attempts to address some of these issues, the use of redundant systems significantly increases the required size, weight, and power (SWAP) of the system. The present invention has the potential to reduce SWAP by removing the need for redundancy.

[0007] Therefore, there is a continuing need for a perception system that provides semantic information and error-corrected machine understanding while reducing SWAP without redundancy. Summary of the invention

[0008] The present disclosure provides a system for detecting and correcting perception errors in a perception system. In one aspect, the system includes a memory and one or more processors. The memory is a non-temporary computer-readable medium, and the non-temporary computer-readable medium is encoded with executable instructions, so that when the instructions are executed, the one or more processors perform several operations, such as: generating a list of detection targets based on the perception data of the scene; generating a list of background classes based on the background associated with the list of detection targets in the perception data; for each detection target in the list of detection targets, identifying the nearest background class from the list of background classes; determining the target embedding vector of the target class; determining the background class embedding vector of the nearest background class; and determining semantic features based on the distance between the target embedding vector and the background class embedding vector.

[0009] On the other hand, the system performs the following operations: generating a probability distribution of the semantic features, the probability distribution having a true positive distribution and a false positive distribution; identifying a lower bound and an upper bound of the true positive distribution so that a range between the lower bound and the upper bound represents a confidence probability P of a true positive probe TP , so that the confidence probability P TP is an axiom of input perception parameters; adjusting the input perception parameters based on the axiom to generate optimal perception parameters; and adjusting one or more perception parameters of the perception system based on the optimal perception parameters.

[0010] In another aspect, the semantic feature is a cosine similarity metric.

[0011] Furthermore, the semantic features are conditional random field (CRF) features, where co-occurrence statistics are obtained through a probabilistic framework, where maximum a posteriori inference is used to determine the likelihood of co-occurrence targets.

[0012] In another aspect, the system performs the following operations: causing the autonomous vehicle to initiate a physical operation based on the optimal perception parameters.

[0013] Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium, and the computer-readable instructions can be executed by a computer having one or more processors, so that when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented method includes the act of causing a computer to execute these instructions and perform the resulting operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the present invention, taken in conjunction with the following drawings, in which:

[0015] Figure 1 is a block diagram depicting components of a system according to various embodiments of the present invention;

[0016] Figure 2 is an illustrative diagram of a computer program product embodying aspects of the present invention;

[0017] Figure 3 is an illustrative diagram depicting prerequisites for generating semantic features according to various embodiments of the present invention;

[0018] Figure 4 is a flow chart depicting a semantic feature generation process according to various embodiments of the present invention;

[0019] Figure 5 is an illustration diagram depicting the structure of a conditional random field (CRF) for five different example objectives according to various embodiments of the present invention;

[0020] Figure 6 is an illustration diagram depicting example misclassifications in object detection according to various embodiments of the present invention;

[0021] Figure 7 is a flowchart depicting an overview of a PSTL framework according to various embodiments of the present invention;

[0022] Figure 8 is a graph depicting sample probability distributions of probes according to various embodiments of the present invention. DETAILED DESCRIPTION

[0023] The present invention relates to a perceptual system that uses semantic features to detect and correct perceptual errors. The following description is presented to enable one of ordinary skill in the art to make and use the invention and incorporate it into the context of a particular application. Various variations and multiple uses for different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of aspects. Therefore, the present invention is not intended to be limited to the aspects presented, but is consistent with the widest scope consistent with the principles and novel features disclosed herein.

[0024] In the following detailed description, many specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without being limited to these specific details. In other cases, known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present invention.

[0025] The reader is invited to pay attention to all the documents and files filed with the present specification and open to the public for inspection together with the present specification, and the contents of all these documents and files are incorporated herein by reference. Unless otherwise expressly stated, all the features disclosed in this specification (including any attached claims, abstracts and drawings) can be replaced by alternative features for the same, equivalent or similar purposes, and therefore, unless otherwise expressly stated, each feature disclosed is only an example of a series of equivalent or similar features.

[0026] Before describing the present invention in detail, a list of references is first provided. Next, a description of each major aspect of the present invention is provided. Subsequently, the introduction provides the reader with a general understanding of the present invention. Finally, the specific details of each embodiment of the present invention are provided to enable an understanding of the specific aspects.

[0027] (1) List of incorporated references

[0028] The following references are cited throughout this application. For clarity and convenience, these references are listed herein as a centralized resource for the reader. The following references are incorporated herein by reference as if fully set forth herein. These references are cited in this application by reference to the following corresponding reference numbers:

[0029] 1."Wordnet|A Lexical Database For English".Wordnet.Princeton.Edu,2010.

[0030] 2.Tripathi,Subarna,et al."Using scene graph context to improve imagegeneration."arXiv preprint arXiv:1901.03762(2019).

[0031] 3.Sadigh,Dorsa,and Ashish Kapoor."Safe control under uncertainty withprobabilistic signal temporal logic."(2016).

[0032] 4.Krizhevsky,Alex,Ilya Sutskever,and Geoffrey E.Hinton."Imagenetclassification with deep convolutional neural networks."Advances in neuralinformation processing systems,Vol.2,pp.1097–1105,2012.

[0033] 5.Caesar,Holger,Jasper Uijlings,and Vittorio Ferrari."Coco-stuff:Thing and stuff classes in context."Proceedings ofthe IEEE Conference onComputer Vision and Pattern Recognition.2018.

[0034] 6.Kirillov,Alexander,et al."Panoptic segmentation."Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition.2019.

[0035] 7.Xiong,Yuwen,et al."Upsnet:A unified panoptic segmentation network."Proceedings ofthe IEEE Conference on Computer Vision and PatternRecognition.2019.

[0036] 8.Kirillov,Alexander,et al."Panoptic feature pyramid networks."Proceedings of the IEEE Conference on Computer Vision and PatternRecognition.2019.

[0037] 9.Redmon,Joseph,and Ali Farhadi."Yolov3:An incremental improvement."arXiv preprint arXiv:1804.02767(2018).

[0038] 10.He,Kaiming,et al."Mask r-cnn."Proceedings of the IEEEinternational conference on computer vision.2017.

[0039] 11.Chen,Liang-Chieh,et al."Semantic image segmentation with deepconvolutional nets and fully connected crfs."arXiv preprint arXiv:1412.7062(2014).

[0040] 12.Speer,Robert,Joshua Chin,and Catherine Havasi."Conceptnet 5.5:Anopen multilingual graph of general knowledge."Thirty-First AAAI Conference onArtificial Intelligence.2017.

[0041] 13. Mikolov, Tomas, et al. "Distributed representations of words and phrases and their compositionality." Advances in neural information processing systems. 2013.

[0042] 14. Bojanowski, Piotr, et al. "Enriching word vectors with subword information." Transactions of the Association for Computational Linguistics 5 (2017): 135 - 146.

[0043] 15. Joulin, Armand, et al. "Bag of tricks for efficient text classification." arXiv preprint arXiv:1607.01759 (2016).

[0044] 16. U.S. Application No. 17 / 030,354, filed September 23, 2020, entitled, "System And Method Of Perception Error Evaluation And Correction By Solving Optimization Problems Under The Probabilistic Signal Temporal Logic Based Constraints".

[0045] 17. Ren, Shaoqing, et al. "Faster r - cnn: Towards real - time object detection with region proposal networks." Advances in neural information processing systems. 2015.

[0046] (2) Main aspects

[0047] Various embodiments of the present invention include three "main" aspects. The first main aspect is a perception system that uses semantic features to detect and correct perception errors. The system is typically in the form of computer system operating software or in the form of a "hard-coded" instruction set. The system can be incorporated into a variety of devices that provide different functions. The second main aspect is a method that is typically in the form of software that operates using a data processing system (computer). The third main aspect is a computer program product. A computer program product typically represents computer-readable instructions stored on a non-temporary computer-readable medium such as an optical storage device (e.g., a compact disk (CD) or a digital versatile disk (DVD)) or a magnetic storage device (e.g., a floppy disk or tape). Other non-limiting examples of computer-readable media include: a hard disk, a read-only memory (ROM), and a flash memory. These aspects will be described in more detail below.

[0048] Figure 1 A block diagram depicting an example of a system of the present invention (i.e., computer system 100) is provided in . The computer system 100 is configured to perform calculations, processes, operations, and / or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software programs) that reside in a computer-readable memory unit and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behaviors, as described herein. In various aspects, the computer system 100 can be specifically implemented in any device that operates to perform the functions described herein that are suitable for a specific application, such as a desktop computer, a mobile phone or smart phone, a tablet computer, a computer specifically implemented in a mobile platform, or one or more other devices that can execute instructions individually and / or collectively to perform related operations / processing.

[0049] Computer system 100 may include an address / data bus 102 configured to transmit information. In addition, one or more data processing units (such as processor 104 (or multiple processors)) are connected to address / data bus 102. Processor 104 is configured to process information and instructions. In one aspect, processor 104 is a microprocessor. Alternatively, processor 104 can be a different type of processor, such as a parallel processor, an application specific integrated circuit (ASIC), a programmable logic array (PLA), a complex programmable logic device (CPLD), or a field programmable gate array (FPGA), or any other processing component that can be operated to perform related operations.

[0050] The computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled to the address / data bus 102, wherein the volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 may also include a non-volatile memory unit 108 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory, etc.) coupled to the address / data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit such as in “cloud” computing. In one aspect, the computer system 100 may also include one or more interfaces, such as interface 110, coupled to the address / data bus 102. The one or more interfaces are configured to enable the computer system 100 to connect to other electronic devices and computer systems. The communication interface implemented by the one or more interfaces can include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology.

[0051] In one aspect, the computer system 100 may include an input device 112 coupled to the address / data bus 102, wherein the input device 112 is configured to transmit information and command selections to the processor 104. According to one aspect, the input device 112 is an alphanumeric input device (such as a keyboard), which may include alphanumeric keys and / or function keys. Alternatively, the input device 112 may be an input device other than an alphanumeric input device. In one aspect, the computer system 100 may include a cursor control device 114 coupled to the address / data bus 102, wherein the cursor control device 114 is configured to transmit user input information and / or command selections to the processor 104. In one aspect, the cursor control device 114 is implemented using a device such as a mouse, a trackball, a trackpad, an optical tracking device, or a touch screen. Notwithstanding the foregoing, in one aspect, the cursor control device 114 is directed and / or enabled via input from the input device 112, such as in response to using special keys and key sequence commands associated with the input device 112. In an alternative aspect, cursor control device 114 is configured to be managed or directed by voice commands.

[0052] In one aspect, computer system 100 may also include one or more optional computer usable data storage devices, such as storage device 116 coupled to address / data bus 102. Storage device 116 is configured to store information and / or computer executable instructions. In one aspect, storage device 116 is a storage device such as a magnetic or optical drive (e.g., a hard disk drive ("HDD"), a floppy disk, a compact disk read only memory ("CD-ROM"), a digital versatile disk ("DVD")). According to one aspect, display device 118 is coupled to address / data bus 102, wherein display device 118 is configured to display video and / or graphics. In one aspect, display device 118 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphic images and alphanumeric characters recognizable by a user.

[0053] The computer system 100 presented herein is an example computing environment according to one aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, one aspect provides a computer system 100 representing a class of data processing analysis that can be used according to various aspects described herein. In addition, other computing systems can also be implemented. In fact, the spirit and scope of the present technology are not limited to any single data processing environment. Therefore, on the one hand, one or more operations of various aspects of the present technology are controlled or implemented using computer executable instructions (such as program modules) executed by a computer. In one implementation, such a program module includes routines, programs, objects, components and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, one aspect provides one or more aspects of the present technology implemented by utilizing one or more distributed computing environments, such as, in this computing environment, tasks are performed by remote processing devices linked by a communication network, or such as, in this computing environment, various program modules are located in local and remote computer storage media including memory-storage devices.

[0054] Figure 2, a diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in . The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term "instructions" as used in connection with the present invention generally indicates a set of operations to be performed on a computer, and may represent a fragment of an entire program or a single separate software module. Non-limiting examples of "instructions" include computer program code (source or target code) and "hard-coded" electronic devices (i.e., computer operations encoded into a computer chip). "Instructions" are stored on any non-transitory computer-readable medium, such as in a computer's memory or on a floppy disk, CD-ROM, and flash drive. In any case, the instructions are encoded on a non-transitory computer-readable medium.

[0055] (3) Introduction

[0056] As described above, the present disclosure is dedicated to a perception system. The present disclosure provides a system and method for generating features based on semantic information and using such information to detect and correct errors in a perception system. The processing enables the creation of feature embedding vectors in a semantic space. The embedding can encode information cues, including but not limited to target co-occurrence, spatial relationships, target / background taxonomies, word ontology, context based on scene graphs, and etymological relationships. Understanding of word ontology and context based on scene graphs can be found in references No. 1 and No. 2, respectively. The ability to encode such information in an embedding vector helps the machine understand the context in the perception system, rather than relying purely on appearance-based features. The disclosed system utilizes such semantic features using a probabilistic signal temporal logic framework to detect and correct perception errors, including but not limited to target misclassification, target missed detection, target trajectory interruption, and false positive target detection. Understanding of probabilistic signal temporal logic can be found in reference No. 3. The system can use semantic information to create one or more constraints for a probabilistic signal temporal logic framework, and the semantic information is used to detect and correct perception errors in such a perception system.

[0057] The disclosed system is more robust and computationally cheaper than the prior art in detecting and correcting perceptual errors, and importantly, is high performance for basic applications. The disclosed system provides several unique advantages over the prior art, including: (1) conversion of contextual semantic information from images and other sensor outputs into machine-understandable feature embedding vectors, (2) use of semantic feature embeddings to create constraints and axioms in a probabilistic temporal logic framework, (3) improved evaluation and correction of perceptual errors aided by semantic context in a formally verifiable manner, and (4) formulation of a dependency model between objects in a scene, thereby allowing the use of dependency models to improve detection accuracy.

[0058] Those skilled in the art will appreciate that the system has several applications using perception systems. Some non-limiting examples include use in intelligence, surveillance and reconnaissance applications (ISR), autonomous vehicles and other unmanned aerial systems, and target identification and tracking. The system described in the present invention can also be used to benchmark other perception systems. Specific details are provided below.

[0059] (4) Specific details of each implementation method

[0060] The present disclosure describes a method and system for generating feature embeddings that encode semantic information and using such feature embeddings to detect and correct errors. The system first generates semantic features to be used as probes. Using these probes, the system builds a probabilistic signal temporal logic (PSTL), which provides axioms. Using these axiom-based constraints, an optimization problem is solved to synthesize controls for reducing perception errors for a perception system.

[0061] As described above, the system first generates semantic features to be used as probes. Figure 3As shown, there are some prerequisites for computing semantic feature embedding. For example, based on the original input perception data 300 (e.g., an image, etc.), the system must first obtain a list 302 of detected targets in the image and the locations of the detected targets (via instance extraction) and a list 306 of background classes (defined as "stuff" in COCO-stuff, see reference No. 5) together with image masks of these background classes (via background extraction). The list of targets / classes detected in the image is provided by the perception system. The system of the present disclosure is proficient in the perception system (i.e., works with the perception system); in other words, the system of the present disclosure uses the output of the perception system and confirms / improves the results of the perception system based on additional semantic context. The system of the present disclosure can generate semantic features for any target class based on existing domain knowledge, but in one aspect, the system is not responsible for detecting these targets in the image.

[0062] These prerequisites can be calculated based on the original input sensory data 300, such as electro-optical sensor images, lidar depth maps, radar detections, or any combination thereof. For the convenience of this description, images from a camera will be used. For camera images, these prerequisites can be calculated using the state-of-the-art panoramic segmentation 304 model (see references No. 6, No. 7, and No. 8). Alternatively, any good target detection technique (e.g., see references No. 8, No. 9, and No. 10) can be used in conjunction with a semantic segmentation technique (see reference No. 11) to obtain the same prerequisites, which include the identified target classes associated with each detected target and the image location of each detected target.

[0063] In order to use this generative method, the system requires concept word embeddings. For the convenience of this description and as a non-limiting example, ConceptNet Numberbatch is used (see Reference No. 12).

[0064] The following pseudo code can be followed to generate semantic features for target-background relations:

[0065] detected_objects=List of detected objects and locations

[0066] background_classes=List of detected background classes and their location

[0067] for each object in detected_objects

[0068] background_class=findClosestBackgroundClass(object.location,

[0069] background_classes)

[0070] object_embedding=concept_embedding(object.class)

[0071] background_embedding=concept_embedding(background_class)

[0072] semantic_feature=cosineSimilarity(object.class,background_class)

[0073] Figure 4 The processing of generating semantic features is further illustrated. For each detection target 400 from the list 302 of targets, the system finds the nearest background 402 from the list 306 of background classes by using the minimum Euclidean distance from the target pixel to the background pixel. This is done by comparing the detection target 400 with each background class in the list 306 of background classes to identify the background class 404 with the minimum distance. It should be noted that the previous example is not the only way to calculate the nearest background class. Any method for calculating the nearest background class can be used as part of the processing of the present disclosure. Cosine similarity is just an example of a measure that can be used as a feature. It is only used as an example because the present invention is also applicable when using different similarity measures or creating different types of semantic features other than similarity.

[0074] Using the concept word embeddings, the system then proceeds to find the embedding vectors of the target class 408 and the nearest background class 404 via the target embedding 410 and the background embedding 406, respectively. The system then determines the cosine similarity 412 as a similarity measure between the embedding vector 410 of the target 408 and the embedding vector 406 of the background class 404, respectively, using the following formula:

[0075]

[0076] Where A represents the target class and B represents the corresponding nearest background class. θ is the angle between two vectors A and B in high-dimensional space. The cosine of θ is an indicator of the similarity between vectors A and B.

[0077] The calculated cosine similarity 412 can be used as a semantic feature 414. This feature 414 should encode whether the targets are closely related by providing a similarity score between -1 and 1. Since concept embeddings are used instead of simple word embeddings (such as word2vec or fasttext, see references No. 13 to No. 15), the embeddings already encode semantic information instead of just using language context.

[0078] Similar to how the semantic feature of target-background coherence is computed, the system can also compute target-target coherence. If the co-existence of two targets is reasonable, they can be said to be coherent. For example, if a handbag is detected next to a person in an image, it can be said to be reasonable because people carry handbags. Whereas, if a handbag is detected next to a traffic light, the result of the perception system may be questionable. In the current case, to compute the semantic feature of target-target coherence, the system can compute the cosine similarity of the concept embeddings of two target classes that are located close to or adjacent to each other in the source image.

[0079] In the system of the present disclosure, the system uses conditional co-occurrence statistics between targets to further evaluate potential errors in target detection. "Conditional" is because it is considered that if there is no computational overkill and "co-occurrence statistics" to learn the dependencies between targets, the relationship between all semantic targets of a given scene will be redundant. To this end, a conditional random field (CRF) framework is formulated, in which co-occurrence statistics are obtained through a probabilistic framework in which the possibility of co-occurring targets is evaluated conditioned on the scene / scenario using maximum a posteriori probability (MAP) inference. The goal is to build a graphical model that describes the correlation between targets in a scenario and make predictions conditioned on the scenario. For more details, consider a graphical model G = (V, E), where V is a set of M nodes, each corresponding to a specific target. E is the set of edges between cliques. Cliques can be defined using any combination of any number of targets; however, inferences can quickly become intractable. For example, Figure 5 The range of possible clusters in a particular scenario is illustrated. The goal here is to characterize each model using mutual information within the targets (e.g., information about what different targets are like with each other and conditioned on a particular scenario (e.g., an urban driving zone). Mutual information is also not limited to commonality. The term "mutual" is used generically because sometimes patterns are not common, but rather grow consistently naturally between two targets; the statistical consistency is called mutual information.

[0080] This can be written as,

[0081]

[0082] Among them, the clique potential Φ c (y, x) is a non-negative real value obtained from the characteristic function of each group, and where p is the conditional probability, x and y are semantic features of the target pair in the image. If the group is two, they are called a pair; for cases beyond pair analysis, the amount of calculation grows exponentially. In theory, if C is all possible groups, it will be all combinations of targets; but in practice, such inference is intractable. Therefore, the process adheres to pair analysis; however, for the sake of generality, the process is described as a group to illustrate that it is not limited to pairs.

[0083] The feature function defines the connection between nodes. As a non-limiting example and with reference to the experiments described herein, since the process focuses on pairwise interactions, there are two feature functions to be designed: 1) a unary feature function, which is calculated using the confidence of the target detection (before the semantic features are generated); and then (2) a pairwise feature function, which is calculated using the cosine similarity between the semantic features. The number of feature functions that each cluster c has is a design choice. On the one hand, it is desirable to ensure that the feature functions can capture distinctive information, and on the other hand, it is desirable to ensure that the process does not introduce too many parameters to cause overfitting or make the calculation difficult to handle. Therefore, the upper bound is determined in the following manner, that is, if the maximum number of targets in the cluster is N and the minimum number is 1, then the lower bound is N, and the upper bound is all combinations of 2 plus all combinations of 3...+N!. Each cluster potential is obtained by a set of feature functions is factorized, where k is the index of the feature function in cluster c. Usually, the significance of each feature function is is learned based on co-occurrence statistics (i.e., similar to how handbags and people are often detected close to each other in urban scenes), however, in this framework, the distance measure is at the conceptual level. Therefore, each feature function Has a weight factor or "significance" associated with it The higher the weight, the more discriminative the pair is in classifying a given object in the scene.

[0084] Finally, to make a probability measure, a normalization term Z(x) (also known as the partition function defined for all possible groups) is used as follows:

[0085] Z(x;θ)=Σ y Π c∈C {Φ c(y,x;θ)}.

[0086] In this framework, the weights of the feature functions are obtained directly from the normalized co-occurrence statistics (which can potentially be obtained using the normalized cosine distance). In standard CRF terminology, the potential is written as:

[0087]

[0088] Each parameter θ k The value of is directly affected by the sparsity of the data (the data used for training or potentially from existing knowledge, such as from ConceptNet). Increasing sparsity improves the expressiveness of the CRF structure, which is also the basic theory of the Hidden Markov Model (HMM) or Dynamic Bayesian Network (DBN).

[0089] Once the parameters are trained (based on co-occurrence probabilities), MAP inference is applied to each test sample to select the label whose parameters return the highest conditional likelihood. For example, under the condition that the scene is an urban object, the objects "car" and "bus" will have a higher correlation than "car" and "aircraft".

[0090] The generated semantic features can be used to confirm the target-background coherence. Figure 6 As shown, aircraft 600 is misclassified as a surfboard. In this case, using semantic features may be beneficial. The closest background classes for the surfboard detected in the image are sky and road. Using this information, the system can calculate semantic features based on target-background coherence. If the target detection network can provide additional potential classes / categories for the detected target, the system can also calculate semantic features for these classes. In this example, the same semantic features for the aircraft are shown in the table below.

[0091]

[0092]

[0093] Based on the calculated semantic feature values ​​in the table above, it is clear that the surfboard and the sky are not close in the semantic space and are completely different. Based on this observation alone, there is a clear signal from the semantic feature that the surfboard may be an incorrect detection. The latest computer vision technology adopts a candidate generation approach, in which a first target candidate is generated and then the candidate is evaluated and classified. In an attempt to correct the error using the semantic feature, the system can calculate the semantic features of the candidate category of the surfboard bounding box candidate. This is shown in the table above. The system can compare the semantic features from the combination of the aircraft and the sky and road background classes with the semantic features from the combination of the surfboard and background classes. Looking at the values ​​in the table, it is clear that the aircraft is significantly more relevant to the sky and the road than the surfboard. Using this information, the detected error can be fixed to have the correct category using the candidate category from the regional proposal network. For a discussion of regional candidate networks, see reference No. 17.

[0094] like Figure 7 As shown, the invention disclosed herein uses the Probabilistic Signal Temporal Logic (PSTL) 702 framework (see reference No. 16 for PSTL) to use the above semantic features. Figure 4 414 in the figure) is generated as part of the perception probe generation 700 process and is used as a probe. The PSTL framework 702 uses the generated probes, which include but are not limited to object size, aspect ratio, contrast, and entropy. Therefore, semantic features will be used as probes in PSTL 702. For each probe, statistical analysis is used to generate a true positive probability distribution and a false positive probability distribution based on true positive detections and false positive detections. For example, Figure 8 The sample probability distribution using any appropriately processed probes is depicted, examples of which are described in reference No. 16.

[0095] By analyzing the distribution of these true positives 800 and false positives 802 and their intersections, an upper bound 804 and a lower bound 806 for true positives can be established. In this example, the shaded area represents the confidence probability P of a true positive probe. TP . Refer again Figure 7 , described as Axiom 704 for PSTL, provides:

[0096]

[0097] Among them, Pr(·) is the predicate, y is the actual test result, t s :t e It means s With t e time series between, so f(x, t s :t e ) is the time frame t s :t e The sequence of probes in . In this formula, let x be the probe corresponding to the semantic feature, then y will be the correct detection result of the semantic feature. This means that the probe indicates that the semantic feature detection / recognition is valid. Depending on the probe dimension, the probability function can also be multidimensional. Integrating all available axioms from x provides a "multidimensional range" for corresponding detection or recognition. As mentioned above, the x used here is a probe or semantic feature, not an axiom.

[0098] These axioms 704 can then be used to optimize the ideal perception control parameters 708 that will provide the best true positive to false positive ratio. Thus, the perception control parameters can be adjusted based on the optimization. The adjusted perception parameters can be used to modify the inputs to the perception system, or to modify parameters in the hardware of the perception system, or to modify parameters in a model internal to the perception system. All three of these (input, hardware, and model) are considered part of the perception system.

[0099] The ideal perceptual control parameters 708 are optimized using PSTL constraint-based optimization 706 (see, for example, Reference No. 16). The "perceptual dynamics" must first be constructed in order to set up the optimization problem, as follows:

[0100] x t′ =f t (x t )+g t (u t ),

[0101] Among them, x t is the probe state at time t, x t′ is the predicted probe in the next time frame t′. Here, x is defined as the probe state; it is the semantic feature being used as a probe. t (·) is the state transition function, g t (·) is the control input u t The input function (it should be noted that the state transfer function ft should not be consistent with the probe sequence f or the characteristic function f k c The goal is to achieve the optimal u t to reduce the perceptual error, and in doing so, generate the optimal perceptual parameters 708 (i.e., u OPT). In order to meet this goal, the optimization problem is set as:

[0102]

[0103] where J(·) is the cost function for estimating the perception error. Therefore, minimizing J(·) can achieve the optimal perception module control input. Finally, with two or more PSTL-based constraints f(x t ), g(z t ) etc., the final optimization formula becomes:

[0104]

[0105] Using the above approach, the following sample PSTL axioms can be established for semantic target-background coherence:

[0106]

[0107] In this formula, z is the semantic feature. For semantic features, since the cosine similarity value will always be between -1 and 1, and a cosine similarity of 1 will always be an ideal scenario, the process can be designed to ignore the upper bound and form only a one-sided lower bound constraint. In the sample axiom established above, for an aircraft, the semantic target-background coherence feature value should be at time t s :t e At least 99% of the time, the threshold of semantic features and P TPz will differ from one class to another. TPz is a probability constraint on the bounds of true positives. Z and X represent different constraints. In one example, P TPx It can be the constraint of the aircraft boundary, P TPz It could be a constraint on the boundaries of a handbag, Z and X are identifiers in this context, they could be A, B, C or any other letter or phrase.

[0108] The output of the system is a set of constraints used to identify and correct perception errors. These perception errors can be corrected in systems with a variety of different applications, including but not limited to autonomous urban driving, autonomous flight, intelligence surveillance and reconnaissance, search and rescue. Incorrect detections may cause the autonomous system to follow an undesirable trajectory. Therefore, the output of the system of the present disclosure corrects the erroneous detection by adjusting the perception parameters, which in turn affects the planning and decision making of the tangible physical autonomous system. For example, the system can cause the autonomous system to change its movement to correct the trajectory of travel.

[0109] In other words, the system of the present disclosure provides a set of one or more constraints, such as those cited above. These constraints are used to identify perception errors. Other target candidates in the image are then checked to ultimately correct the identified errors. In one aspect, this error identification and correction is an output of the system described in the present invention, which can be relayed to a planner to influence, modify, and plan better and safer trajectories for the autonomous system.

[0110] As another example, a self-driving vehicle may misclassify a bicycle as a person. In the absence of this system, the self-driving car will expect pedestrians to stay on the sidewalk, but in fact the bicycle is not constrained to the sidewalk. This system can correct this error and accurately notify the self-driving vehicle of the bicycle. Using the system disclosed in the present invention, the self-driving vehicle will take a really different route in the physical world to limit itself from entering the bicycle lane to avoid potential accidents. In doing so, the system can enable the self-driving vehicle to initiate physical operations through other systems in the vehicle (such as accelerators, brakes, or steering) to avoid colliding with the detected target (e.g., bicycle in this example). Therefore, on the one hand, the self-driving vehicle will automatically adjust / modify its trajectory to take into account the results of the method and system as described herein. Those skilled in the art will appreciate that this concept can be extended to a variety of applications, such as unmanned aerial vehicles, robotic equipment in factories, etc.

[0111] Finally, while the invention has been described in terms of several embodiments, one of ordinary skill in the art will readily recognize that the invention may have other applications in other contexts. It should be noted that there may be many embodiments and implementations. Furthermore, the appended claims are in no way intended to limit the scope of the invention to the specific embodiments described above. Additionally, any recitation of "a device for..." is intended to elicit a device-plus-function reading of the elements and claims, and any element not specifically recited with "a device for..." should not be interpreted as a device-plus-function element, even if the claim otherwise includes the term "device." Furthermore, while specific method steps have been described in a particular order, these method steps may be performed in any desired order and fall within the scope of the invention.

Claims

1. A system for detecting and correcting perception errors in a perception system, the system comprising: A memory and one or more processors, the memory being a non-transitory computer-readable medium having executable instructions encoded thereon, such that when executing the instructions, the one or more processors perform the following operations: Generate a list of detection target classes based on the scene's perception data; generating a list of background classes based on background in the perception data associated with the list of detection target classes; For each detection target in the list of detection target classes, identifying a nearest background class from the list of background classes; Determining an object embedding vector of the detected object class; Determining a background class embedding vector for the nearest background class; Determine a semantic feature based on a distance between the target embedding vector and the background class embedding vector; generating a probability distribution of the semantic feature, wherein the probability distribution has a true positive distribution and a false positive distribution; Identify the lower and upper bounds of the true positive distribution so that the region between the lower and upper bounds represents the confidence probability P of a true positive probe. TP , so that the confidence probability P TP is the rule for inputting perception parameters; adjusting the input perception parameters based on the rules to generate optimal perception parameters; as well as One or more perception parameters of the perception system are adjusted based on the optimal perception parameters.

2. The system according to claim 1, wherein: The semantic feature is a cosine similarity measure.

3. The system according to claim 1, wherein: The semantic features are conditional random field (CRF) features, where co-occurrence statistics are obtained through a probabilistic framework, where maximum a posteriori probability inference is used to determine the likelihood of co-occurrence targets.

4. The system of claim 1, further comprising the following operation: causing the autonomous vehicle to initiate a physical operation based on the optimal perception parameter.

5. A computer program product for detecting and correcting perception errors in a perception system, the computer program product comprising: A non-transitory computer-readable medium having executable instructions encoded thereon such that when the instructions are executed by one or more processors, the one or more processors perform the following operations: Generate a list of detection target classes based on the scene's perception data; generating a list of background classes based on background in the perception data associated with the list of detection target classes; For each detection target in the list of detection target classes, identifying a nearest background class from the list of background classes; Determining an object embedding vector of the detected object class; Determining a background class embedding vector for the nearest background class; Determine a semantic feature based on a distance between the target embedding vector and the background class embedding vector; generating a probability distribution of the semantic feature, wherein the probability distribution has a true positive distribution and a false positive distribution; Identify the lower and upper bounds of the true positive distribution so that the region between the lower and upper bounds represents the confidence probability P of a true positive probe. TP , so that the confidence probability P TP is the rule for inputting perception parameters; adjusting the input perception parameters based on the rules to generate optimal perception parameters; as well as One or more perception parameters of the perception system are adjusted based on the optimal perception parameters.

6. The computer program product according to claim 5, wherein: The semantic feature is a cosine similarity measure.

7. The computer program product according to claim 5, wherein: The semantic features are conditional random field (CRF) features, where co-occurrence statistics are obtained through a probabilistic framework, where maximum a posteriori probability inference is used to determine the likelihood of co-occurrence targets.

8. The computer program product of claim 5, further comprising the operation of causing an autonomous vehicle to initiate a physical operation based on the optimal perception parameter.

9. A computer-implemented method for detecting and correcting perceptual errors in a perceptual system, the method comprising the following acts: causing one or more processors to execute instructions encoded on a non-transitory computer readable medium such that, when executing the instructions, the one or more processors perform the following operations: Generate a list of detection target classes based on the scene's perception data; generating a list of background classes based on background in the perception data associated with the list of detection target classes; For each detection target in the list of detection target classes, identifying a nearest background class from the list of background classes; Determining an object embedding vector of the detected object class; Determining a background class embedding vector for the nearest background class; Determine a semantic feature based on a distance between the target embedding vector and the background class embedding vector; generating a probability distribution of the semantic feature, wherein the probability distribution has a true positive distribution and a false positive distribution; Identify the lower and upper bounds of the true positive distribution so that the region between the lower and upper bounds represents the confidence probability P of a true positive probe. TP , so that the confidence probability P TP is the rule for inputting perception parameters; adjusting the input perception parameters based on the rules to generate optimal perception parameters; as well as One or more perception parameters of the perception system are adjusted based on the optimal perception parameters.

10. The method according to claim 9, wherein: The semantic feature is a cosine similarity measure.

11. The method according to claim 9, wherein: The semantic features are conditional random field (CRF) features, where co-occurrence statistics are obtained through a probabilistic framework, where maximum a posteriori probability inference is used to determine the likelihood of co-occurrence targets.

12. The method according to claim 9 further comprises the following operation: causing the autonomous vehicle to initiate a physical operation based on the optimal perception parameter.

Images