Classification process evaluation

Abstract

The present disclosure provides a method for classification process evaluation, the method comprising: (a) receiving, at processing circuitry, classification data generated by a classification process for testing an enhanced version of a sensing information unit; (b) evaluating the classification data across the enhanced version by analyzing a distribution of at least selected classification values of the classification data; (c) determining a compatibility of the classification process with respect to the received classification data based on the evaluation; and (d) issuing a compatibility indication in accordance with the determined compatibility, the compatibility indication indicating compatibility of a classification process to classify the element captured by the sensing information unit.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method for evaluating an image classification process, a non-transitory computer-readable storage medium, and a computer-implemented system. Background Technology

[0002] Assisted and autonomous driving systems are known in the art. In such systems, a computer-implemented system controls (at least to some extent) some or all of the driving functions of a vehicle, such as speed, telemetry, braking, etc. Vehicles are typically equipped with one or more sensors to provide the system with sensed information about the driving environment. The driving system typically uses the sensed information about the driving environment to determine how to drive on the road.

[0003] One of the main tasks related to driving is classification.

[0004] Sensing information can be acquired and / or processed under different conditions (e.g., different sensing conditions and / or different sensing information processing parameters and / or noise).

[0005] Therefore, there is an increasing need to provide a robust classification system that can provide consistent classification results regardless of different conditions. Summary of the Invention

[0006] This disclosure provides a method for evaluating an image classification process, a non-transitory computer-readable storage medium, and a computer-implemented system.

[0007] In a first aspect of this disclosure, a computer-implemented method for evaluating a classification process includes receiving, at a processing circuit, classification data generated by an enhanced version of a classification process for testing a sensing information unit; evaluating the classification data across the enhanced version by analyzing the distribution of selected classification values ​​of the classification data; determining, based on the evaluation, the compatibility of the classification process with respect to the received classification data; and issuing a compatibility indication based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the testing sensing information unit.

[0008] In another aspect of this disclosure, a non-transitory computer-readable medium for evaluating a classification process stores instructions that, once executed by a computerized system, cause the computerized system to: receive classification data generated by an enhanced version of a classification process for testing a sensing information unit; evaluate the classification data across the enhanced version by analyzing the distribution of selected classification values; determine the compatibility of the classification process with respect to the received classification data based on the evaluation; and issue a compatibility indication based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the testing sensing information unit.

[0009] In another aspect of this disclosure, a computerized system for evaluating a classification process includes: a memory unit configured to store classification data generated by an enhanced version of a classification process used to test a sensing information unit; and processing circuitry configured to: evaluate the classification data across the enhanced version by analyzing the distribution of selected classification values; determine the compatibility of the classification process with respect to the received classification data based on the evaluation; and issue a compatibility indication based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the testing sensing information unit.

[0010] It should be understood that all combinations of the foregoing and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0011] The embodiments of this disclosure will be more fully understood and appreciated through the following detailed description, in conjunction with the accompanying drawings, in which:

[0012] Figure 1A A block diagram of a computerized system within a vehicle according to some embodiments of the present disclosure is shown;

[0013] Figure 1B This is a block diagram of a computerized classification system in a vehicle according to some embodiments of the present disclosure;

[0014] Figure 2 This is a flowchart illustrating an exemplary method for classifying data and determining the compatibility of the classification according to some embodiments of the present disclosure;

[0015] Figure 3 This is a block diagram of a system for performing classification and determining the compatibility of classifications according to some embodiments of this disclosure;

[0016] Figure 4These are examples of sensing information according to some embodiments of the present disclosure, which includes a sensed image and various information elements associated with the sensed image;

[0017] Figure 5 Examples of sensing information according to some embodiments of this disclosure include a sensed image and key points and cropped images associated with the sensed image; and

[0018] Figure 6 These are examples of sensing information according to some embodiments of the present disclosure, which includes a sensed image and a cropped image associated with the sensed image and various information elements. Detailed Implementation

[0019] The embodiments of this disclosure will be described in detail with reference to the accompanying drawings, focusing on the following technical problems, structural features, objectives, and effects. Specifically, the terminology used in the embodiments of this disclosure is for the purpose of describing certain embodiments only and is not intended to limit the disclosure. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, those skilled in the art will understand that this disclosure can be practiced without these specific details. In other instances, well-known methods, processes, and components have not been described in detail so as not to obscure the disclosure. The subject matter of this disclosure is particularly pointed out and clearly claimed in the concluding section of this specification. However, the organization and operation of this disclosure, as well as its objectives, features, and advantages, can be best understood by referring to the following detailed description when read in conjunction with the accompanying drawings. Because the embodiments shown in this disclosure can be implemented largely using electronic components and circuits known to those skilled in the art, the explanation of details will not exceed the necessary scope described above in order to understand and grasp the basic concepts of this disclosure and to avoid obscuring or diverting from the teachings of this disclosure. For example, the specification and / or drawings may relate to a processor or processing circuitry. A processor may be processing circuitry. The processing circuitry can be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), fully custom integrated circuits, or combinations of such integrated circuits.

[0020] The following descriptions and / or accompanying figures may refer to images or image frames. Images are examples of media units. Any references to images may be modified as necessary to suit media units. Media units may be examples of Sensing Information Units (SIUs). Any references to media units may be modified as necessary to suit any type of natural signal, such as, but not limited to, signals generated naturally, signals representing human behavior, signals representing operations related to vehicle signals, geodetic signals, geophysical signals, text signals, numerical signals, time-series signals, etc. Any references to media units may be modified as necessary to suit SIUs. SIUs can be of any type and can be sensed by any type of sensor, such as visual light cameras, audio sensors, sensors capable of sensing infrared light, radar imaging, ultrasound, electro-optics, radiography, light detection and ranging (LIDAR), thermal sensors, passive sensors, active sensors, etc. Sensing may include generating samples (e.g., pixels, audio signals, etc.) representing signals transmitted or otherwise arriving at the sensor. SIUs may have one or more images, one or more video clips, text information about one or more images, text describing kinematic information, etc.

[0021] Any combination of any modules or units listed in any drawing, any part of the specification, and / or any claim may be provided. Any unit and / or module shown in the application may be implemented in hardware and / or stored in non-transitory computer-readable media as code, instructions, and / or commands, and may be included in a vehicle, outside a vehicle, in a mobile device, a server, etc. A vehicle may be any type of vehicle, such as a ground transport vehicle, an air transport vehicle, or a water transport vehicle. A vehicle is also referred to as an ego-vehicle. It should be understood that autonomous driving includes at least partially autonomous (semi-autonomous) driving of a vehicle, which includes all Level 2 types or higher as defined in the SAE standard.

[0022] The present invention provides a method, system, and computer-readable medium that are robust and configured to provide robust classification regardless of changes in one or more conditions associated with the acquisition and / or processing in a sensing information unit.

[0023] According to one embodiment, classification values ​​associated with an augmented version of the sensing information unit are evaluated to determine whether the classification process provides consistent classification values, regardless of the changes introduced by the augmentation.

[0024] According to one embodiment, the classification process is evaluated during a testing phase, which follows the training phase but can precede inference. Identifying inadequate classification processing during the testing phase—particularly before inference—reduces classification errors that could lead to accidents.

[0025] refer to Figures 1A to 1B The diagram shows a vehicle 100 and a remote computerized system 134 that may be located outside the vehicle. The vehicle 100 includes a sensing system 110, a communication system 130, one or more memory and / or storage units 120, a network 132, a control unit 125, and a processing system 124 having a processor 126, which includes a plurality of processing circuits 126(1)-126(J).

[0026] One or more memory and / or storage units 120 are shown storing an operating system 194, software 193 (especially software required to perform method 200), information 191, and metadata 192 (especially information and metadata required to perform method 200). Information may include environmental information. Metadata may include (especially in relation to the execution of method 200) any metrics or results of processed information.

[0027] Network 132 communicates with vehicles and remote computerized systems 134 such as servers and cloud computers.

[0028] The control unit 125 is configured to control various operations related to the vehicle, such as, but not limited to, the various steps of method 200.

[0029] One or more memory and / or storage units 120 are shown storing an operating system 194, software 193 (especially software required for executing method 200), information 191, and metadata 192 (especially information and metadata required for executing method 200). Information may include environmental information. Metadata may include (especially in relation to the execution of method 200) any metrics or results of processed information.

[0030] Figure 1B and Figure 1A The difference lies in the inclusion of additional units, such as ADAS control unit 123, autonomous driving control unit 122, and vehicle computer 121, as well as more examples including contents stored in one or more memories and / or storage units 120.

[0031] The sensing system 110 may include optics, a group of sensing elements, a readout circuit, and an image signal processor. Following the optics is the group of sensing elements, such as a line of sensing elements or an array of sensing elements forming the group. Following the group of sensing elements is the readout circuit, which reads the detection signal generated by the group of sensing elements. The image signal processor is configured to perform initial processing of the detection signal, for example, by improving the quality of the detection information, performing noise reduction, etc. The sensing system 110 is configured to output one or more sensing information units (SIUs).

[0032] ADAS control unit 123 is configured to control ADAS operation.

[0033] The autonomous driving control unit 122 is configured to control the autonomous driving of autonomous vehicles.

[0034] The vehicle computer 121 is configured to control the operation of the vehicle, particularly the engine, transmission and any other vehicle systems or components.

[0035] The processing system 124 may include processor 126 and one or more other processors, and is configured to perform any of the methods described in the specification.

[0036] According to one embodiment, a computerized system is provided, including a memory unit and processing circuitry (e.g., processor 126). The memory unit is configured to store classification data generated by an enhanced version of the classification process used to test sensing information units.

[0037] The processing circuit is configured to evaluate the classification process of the neural network by evaluating an enhanced version of the classification data through analysis of the distribution of at least selected classification values ​​of the classification data.

[0038] The selected categorical values ​​can be all categorical values ​​or only a subset of them. Selection can be made in any manner—for example, selection can be applied iteratively, where a first set of categorical values ​​is selected and evaluated (by performing the next step of the evaluation process)—where the absence of statistically significant categorical values ​​from the first set can be considered an incompatibility of the classification process—without needing to evaluate a second set of categorical values. The first set can include 10%, 20%, 30%, 40%, 50%, or 60% of the categorical values.

[0039] According to one embodiment, the enhanced version is generated by one or more machine learning processes and / or includes applying at least one of different enhancement parameters, such as at least one cropping parameter, at least one color condition, at least one warping condition, and at least one distortion condition. For example, by performing random cropping of the image and feeding back only a portion of the cropped image (e.g., 70%, 80%, or 90% of the cropped image) to the network.

[0040] According to one embodiment, the processing circuit analyzes the distribution using at least one of the following criteria:

[0041] a) Determine whether all classification values ​​are equal to each other (indicating the presence of identical elements within the enhanced version of the test sensing information unit).

[0042] b) Determine whether all classification values ​​are similar to each other. Similarity means that they identify the same elements or identify elements that are defined as similar to each other—for example, different types of pedestrians (e.g., different ages, different sizes, etc.) can be defined as similar to each other, and different types of vehicles (e.g., vehicles from the same manufacturer, vehicles with the same purpose—where different purposes can refer to SUVs, vans, buses, family cars, etc.) can be defined as similar to each other.

[0043] c) Determine whether the categorical values ​​of at least one defined percentile are equal to each other (or at least similar to each other).

[0044] d) Determine whether at least a number of categorical values ​​are equal to each other (or at least similar to each other).

[0045] e) Determine whether the number of different category values ​​is below a defined threshold.

[0046] f) Determine whether the categorical value is statistically significant.

[0047] According to one embodiment, the processing circuitry is configured to determine the compatibility of the classification process with respect to the received classification data based on an evaluation.

[0048] According to one embodiment, the determination includes evaluating any one or any other criterion among the criteria (a)-(f) listed above.

[0049] According to one embodiment, the processing circuitry is configured to determine that the classification process is compatible with the received classification data when one or more criteria are met (e.g., whether all classification values ​​(or at least the defined number) are equal to each other, referring to criteria (a) and (c)). This also applies to any other criteria (b) and (d)-(f).

[0050] According to one embodiment, the processing circuitry is configured to respond to the determination.

[0051] According to one embodiment, the response includes at least one of the following:

[0052] a) Issue a compatibility indication based on the determined compatibility, which indicates the compatibility of the classification process used to classify elements captured by the sensing information unit.

[0053] b) Triggering the assignment of another classification process to classify the test sense information based on the determined compatibility.

[0054] c) Routing the enhanced version of the test sensing information unit to another classification process.

[0055] d) Another classification process that assigns test sensing information based on the determined compatibility.

[0056] e) Determining the driving-related actions to be performed by the vehicle. Driving-related actions can be fully autonomous actions, such as maintaining the current progress of the vehicle, changing the progress of the vehicle, where progress refers to at least one of the vehicle's direction of travel, speed of travel, and acceleration. Alternatively, driving-related actions are ADAS actions, such as driving scheme actions that suggest the performance of more restricted driving-related actions (restricted in the sense of their duration).

[0057] f) Perform driving-related operations.

[0058] g) Trigger the execution of driving-related operations.

[0059] h) Determine the navigation route for the vehicle.

[0060] i) Trigger to determine the navigation path of the vehicle.

[0061] j) Send information about the classification to one or more AI agents.

[0062] k) Triggers the sending of information about the classification to one or more AI agents.

[0063] l) Activate one or more AI agents based on classification.

[0064] m) Triggers the activation of one or more AI agents based on classification.

[0065] According to one embodiment, triggering includes transmitting a signal other than a compatibility indication. This signal may be an interrupt request signal, an arbitrator request signal, or any other electronic and / or optical signal.

[0066] According to one embodiment, the processing circuitry performs a classification process to provide classification data.

[0067] According to one embodiment, another processing circuit performs the classification process.

[0068] According to one embodiment, the classification process is an embedding-based classification process.

[0069] Examples of embedding-based classification processes also exist. Figures 4 to 6 As shown in the figure, it will be described in more detail below.

[0070] The various accompanying figures illustrate examples of units and / or software and / or information items and / or steps and / or components. These examples are provided for the sake of brevity of explanation. At least one of the units and / or software and / or information items and / or steps and / or components is optional or mandatory.

[0071] One or more memory and / or storage units 120 are configured to store firmware and / or software, one or more operating systems, data and metadata required to perform any of the methods mentioned in this application.

[0072] Now for specific reference Figure 1B One or more memory and / or storage units 120 store at least some of the following:

[0073] a) Enhanced software 193-1 for generating an enhanced version of the sensing information unit.

[0074] b) Operating system 194.

[0075] c) Classification software 193-2 used to apply one or more classification processes – to be evaluated during testing.

[0076] d) Classification compatibility software 193-3, used to determine whether the classification process is compatible with the received classification data.

[0077] e) The response software 193-4 used to respond to this determination.

[0078] f) Sensing information unit 271, which may be a test sensing information unit.

[0079] g) Enhanced version 272 of the sensing information unit.

[0080] h) Classification value 275(1)-275(M).

[0081] i) Compatibility indicator 279.

[0082] The vehicle computer 121 can communicate with the engine control module, transmission control module, drivetrain control module, etc.

[0083] The memory and / or storage unit 120 is shown as storing software. Any references to the software shall be modified as necessary to suit the code and / or firmware and / or instructions and / or commands, etc.

[0084] According to one embodiment, one or more memory and / or storage units 120 include one or more memory cells, each memory cell including one or more storage banks.

[0085] According to one embodiment, one or more memory and / or storage cells 120 include volatile memory and / or non-volatile memory. The one or more memory and / or storage cells 120 may be random access memory (RAM) and / or read-only memory (ROM).

[0086] According to one embodiment, a non-volatile storage unit is a mass storage device that can provide non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data for a processor or any other unit of a vehicle. For example, but not limited to, a mass storage device can be a hard disk, removable disk, removable optical disk, magnetic tape cassette or other magnetic storage device, flash memory card, CD-ROM, digital versatile disc (DVD) or other optical storage, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0087] Any content can be stored in any part of the memory and / or storage unit or in any type of memory and / or storage unit.

[0088] According to one embodiment, at least one memory unit stores at least one database—such as any database known in the art—such as Access SQL Server MySQL, PostgreSQL, etc.

[0089] Various units and / or components communicate with each other using any communication elements and / or protocols. An example of a communication system is shown as 130. Other communication elements may be provided.

[0090] Now for reference Figure 1A and 1B The communication system 130 can communicate with various processors and / or units as well as the network 132.

[0091] The communication system 130 may include a bus. This represents one or more of several possible bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of the various bus architectures. As examples, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, an Accelerated Graphics Port (AGP) bus, and peripheral component interconnects (PCI), PCI-Express buses, the Personal Computer Memory Card Industry Association (PCMCIA), Universal Serial Bus (USB), etc. This bus and all buses specified in this specification may also be implemented via wired or wireless network connections and by each subsystem.

[0092] Network 132 is located outside the vehicle and is used for communication between the vehicle and at least one remote computing system. As an example, the remote computing system may be a personal computer, laptop computer, portable computer, server, router, network computer, peer-to-peer device, or other public network node. The logical connection between the processor and any remote computing system can be implemented via a local area network (LAN) and a general wide area network (WAN). Such network connections can be implemented via a network adapter (which may belong to communication system 130) in both wired and wireless environments. Such networking environments are common and routine in offices, enterprise-wide computer networks, intranets, and larger networks such as the Internet.

[0093] It should be noted that at least a portion of the contents shown stored in one or more memory / storage units 120 may be stored outside the vehicle. It should also be noted that the processor can evaluate features generated by multiple detectors.

[0094] Reference Figure 2 The diagram illustrates a flowchart of a method 200 for classifying data and determining the compatibility of the classification. At 210, classification data generated by an enhanced version of the classification process used to test the sensing information unit is received by the processing circuit.

[0095] Next, at point 220, the categorical data is evaluated across the enhanced version by analyzing the distribution of at least the selected categorical values. The selected categorical values ​​can be all categorical values ​​or only a subset of them. Selection can be made in any manner—for example, selection can be applied iteratively, where a first set of categorical values ​​is selected and evaluated (by performing the next step of the evaluation process)—where the absence of statistically significant categorical values ​​from the first set can be considered an incompatibility of the classification process—without needing to evaluate a second set of categorical values. The first set can include 10%, 20%, 30%, 40%, 50%, or 60% of the categorical values.

[0096] According to one embodiment, the enhanced version is generated by one or more machine learning processes and / or includes applying at least one of different enhancement parameters, such as at least one cropping parameter, at least one color condition, at least one warping condition, and at least one distortion condition. For example, by performing random cropping of the image and feeding back only a portion of the cropped image (e.g., 70%, 80%, or 90% of the cropped image) to the network.

[0097] According to one embodiment, step 220 may further include analyzing the distribution by at least one of the following criteria:

[0098] a) Determine whether all classification values ​​are equal to each other (indicating the presence of identical elements within the enhanced version of the test sensing information unit).

[0099] b) Determine whether all classification values ​​are similar to each other. Similarity means that they identify the same elements or identify elements that are defined as similar to each other—for example, different types of pedestrians (e.g., different ages, different sizes, etc.) can be defined as similar to each other, and different types of vehicles (e.g., vehicles from the same manufacturer, vehicles with the same purpose—where different purposes can refer to SUVs, vans, buses, family cars, etc.) can be defined as similar to each other.

[0100] c) Determine whether the categorical values ​​of at least one defined percentile are equal to each other (or at least similar to each other).

[0101] d) Determine whether at least a number of categorical values ​​are equal to each other (or at least similar to each other).

[0102] e) Determine whether the number of different category values ​​is below a defined threshold.

[0103] f) Determine whether the categorical value is statistically significant.

[0104] Subsequently, at point 230, the compatibility of the classification process with respect to the received classification data is determined based on the evaluation.

[0105] According to one embodiment, the determination includes evaluating any one or any other criterion among the criteria (a)-(f) listed above.

[0106] According to one embodiment, step 230 may include determining that the classification process is compatible with the received classification data when one or more criteria are met (e.g., whether all classification values ​​(or at least the defined number) are equal to each other, referring to criteria (a) and (c)). This also applies to any other criteria (b) and (d)-(f).

[0107] Furthermore, at step 240, this determination is responded to. According to one embodiment, step 240 may include at least one of the following:

[0108] a) Issue (step 241) a compatibility indication based on the determined compatibility, which indicates the compatibility of the classification process used to classify the elements captured by the sensing information unit.

[0109] b) Triggering the assignment of another classification process to classify the test sense information based on the determined compatibility.

[0110] c) Routing the enhanced version of the test sensing information unit to another classification process.

[0111] d) Another classification process that assigns test sensing information based on the determined compatibility.

[0112] e) Determining the driving-related actions to be performed by the vehicle. Driving-related actions can be fully autonomous actions, such as maintaining the current progress of the vehicle, changing the progress of the vehicle, where progress refers to at least one of the vehicle's direction of travel, speed of travel, and acceleration. Alternatively, driving-related actions are ADAS actions, such as driving scheme actions that suggest the performance of more restricted driving-related actions (restricted in the sense of their duration).

[0113] f) Perform driving-related operations.

[0114] g) Trigger the execution of driving-related operations.

[0115] h) Determine the navigation route for the vehicle.

[0116] i) Trigger to determine the navigation path of the vehicle.

[0117] j) Send information about the classification to one or more AI agents.

[0118] k) Triggers the sending of information about the classification to one or more AI agents.

[0119] l) Activate one or more AI agents based on classification.

[0120] m) Triggers the activation of one or more AI agents based on classification.

[0121] For the sake of brevity, in Figure 2 The responses (a)-(m) are not shown. According to one embodiment, triggering includes transmitting a signal other than a compatibility indication. This signal may be an interrupt request signal, an arbitrator request signal, or any other electronic and / or optical signal.

[0122] According to one embodiment, method 200 includes performing a classification process to provide classification data.

[0123] According to one embodiment, method 200 does not include performing a classification process.

[0124] According to one embodiment, step 210 is performed by a processing circuit that performs the classification process.

[0125] According to one embodiment, the classification process is an embedding-based classification process. An example of an embedding-based classification process is... Figures 4 to 6 As shown in the image.

[0126] Reference Figure 3 System 300 includes a sensing information unit 271, an enhancement unit 272, a classification unit 274, a classification compatibility unit 276, and a response unit 277. System 300 can be configured to perform a process including the following steps:

[0127] a) The enhancement unit 272 receives the sensing information provided by the sensing information unit 271.

[0128] b) Enhanced versions 273(1)-273(K) of the sensing information generated by the enhancement unit 272.

[0129] c) The enhanced versions 273(1)-273(K) of the sensed information generated by the classification unit 274 are classified to provide classification values ​​275(1)-275(M), which indicate the elements captured by the enhanced versions 273(1)-273(K) of the sensed information. The classification unit may include a representation generation unit 274-1 and a representation-based classifier 274-2. The representation may be an embedding, embedded features, or features independent of the embedding.

[0130] d) The classification compatibility unit 276 determines the compatibility of the classification process with respect to the received classification data.

[0131] e) The response unit 277 responds to this determination. Here, the compatibility indication 279 may be issued by the response unit, which includes the routing unit 277-1 and the allocation unit 277-2.

[0132] Figure 3 It also shows one or more other classification units 278 that can perform classification when classification unit 274 is applied to perform an incompatible classification process.

[0133] Any unit is a hardware unit or is implemented by executing instructions through processing circuitry.

[0134] Reference Figure 4 The initial sensing information 400 may include an image 901 having a bounding box 902 indicating the location of an object 903; a cropped image 904 mainly comprising pixels of the object 903; an object embedding 910 of the cropped image; features 912 of the object embedding; and reference embedding feature clusters 920(1)-920(W) represented by reference embedding features 921(1)-921(X) (which may or may not be included in the reference embedding feature clusters), wherein each reference embedding feature cluster may contain more than one reference embedding feature; and those included in Other reference embedding features 922(1)-922(Y) in the reference embedding feature cluster (and different from reference embedding features 921(1)-921(X)); outliers 925(1)-925(Z) located outside the cluster; reference embedding feature cluster metadata (923(1)-923(W)) which provides information such as the object classification of the cluster; and matching reference embedding feature cluster 920(w) which is represented by reference embedding feature 921(w) and associated with metadata 923(w) indicating the object classification.

[0135] According to one embodiment, an object embedding information item is an object embedding, a reference embedding information item is a reference embedding, and a cluster of reference embedding information items is a cluster of reference embeddings.

[0136] According to one embodiment, the cropped sensing information unit essentially consists of information indicating the object. Cropping increases the accuracy of object embedding because most irrelevant information is removed from the cropped sensing information unit.

[0137] According to one embodiment, a clipped sensing information unit is generated based on an initial sensing information unit and a bounding box indicating an object within the initial sensing information unit.

[0138] According to one embodiment, clipped sensing information is generated based on an initial sensing information unit and a plurality of key points associated with an object within the initial sensing information unit. The key points can be found in any known manner. According to one embodiment, the key points are found in the manner shown in U.S. Patent 11,037,015, which is incorporated herein by reference.

[0139] According to one embodiment, clipped sensing information is generated based on an initial sensing information unit and an initial sensing information unit region, the initial sensing information unit region including multiple key points associated with an object within the initial sensing information unit.

[0140] Assuming the transformation module (especially the embedding generator within the transformation module) is configured to process content of a given shape—for example, it is configured to process clipped sensing information units containing relevant information within a rectangular region comprising sensing information units—the method includes defining a rectangular region to include multiple keypoints (the rectangular region can be defined as small as possible—or including a finite amount of information outside the smallest region containing the keypoints)—and processing the content of the rectangular region. This processing may include aligning the rectangular region (which can be oriented to a horizontal line) before providing the rectangular frame to the embedding generator.

[0141] Reference Figure 5 The image 950 shows sensing information 500 having an object 903 including key points 930 of the object and a bounding box 932 surrounding the key points 930, as well as a cropped image 904. The bounding box 932 is oriented together with the object 930, and the cropped image 904 includes an aligned bounding box 933.

[0142] Reference Figure 6 The initial sensing information 600 may include an image 901, which includes a bounding box 902 indicating the location of an object 903; a cropped image 904 mainly comprising pixels of the object 903; an object embedding 910 of the cropped image; a cluster of reference embeddings 970(1)-970(W) represented by reference embeddings 971(1)-971(X), wherein each reference embedding cluster may have more than one reference embedding; other reference embeddings 972(1)-972(Y) included in the reference embedding cluster (and different from reference embeddings 971(1)-971(X)); outliers 975(1)-975(Z) located outside the cluster; reference embedding cluster metadata (973(1)-973(W)) which provides information such as the object classification of the cluster; and a matching reference embedding cluster 970(w) represented by reference embedding 971(w).

[0143] In the foregoing detailed description, numerous specific details have been set forth in order to provide a thorough understanding of this disclosure. However, those skilled in the art will understand that this disclosure can be practiced without these specific details. In other instances, well-known methods, processes, and components have not been described in detail so as not to obscure this disclosure.

[0144] The subject matter of this disclosure is specifically pointed out and clearly claimed in the concluding section of this specification. However, the organization and operation of this disclosure, as well as its purpose, features, and advantages, can be best understood by referring to the following detailed description when read in conjunction with the accompanying drawings.

[0145] It should be understood that, for the sake of simplicity and clarity, the elements shown in the figures need not be drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Furthermore, reference numerals may be repeated in the figures where deemed appropriate to indicate corresponding or similar elements.

[0146] Because the embodiments shown in this disclosure can be largely implemented using electronic components and circuits known to those skilled in the art, the details will not be explained to the extent deemed necessary, as shown above, in order to understand and comprehend the basic concepts of this disclosure and to avoid confusing or deviating from its teachings.

[0147] Any references to the methods in this specification should be modified as necessary to suit the devices or systems capable of performing the methods and / or the non-transitory computer-readable media storing instructions for performing the methods.

[0148] Any references to the system or device in this specification shall be modified as necessary to suit methods that can be executed by the system, and / or may be modified as necessary to suit non-transitory computer-readable media storing instructions that can be executed by the system.

[0149] Any references to non-transitory computer-readable media in this specification shall be modified as necessary to suit a device or system capable of executing instructions stored in a non-transitory computer-readable medium, and / or may be modified as necessary to suit a method for executing the instructions.

[0150] Any combination of any module or unit listed in any drawing, any part of the specification and / or any claim may be provided.

[0151] Any of the conversion module, active learning module, or cluster module, or any other module described herein, may be implemented in hardware and / or stored in a non-transitory computer-readable medium as code, instructions, and / or commands, and may be included in a vehicle, outside a vehicle, a mobile device, a server, etc.

[0152] The means of transport can be any type of transport, such as ground transport, air transport, or water transport.

[0153] This specification and / or accompanying drawings may refer to images. Images are examples of media units. Any references to images may be modified as necessary to suit the media unit. Media units may be examples of sensed information. Any references to media units may be modified as necessary to suit any type of natural signal, such as, but not limited to, signals generated naturally, signals representing human behavior, signals representing stock market-related operations, medical signals, financial sequences, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, etc. Any references to media units may be modified as necessary to suit sensed information. Sensed information can be of any type and can be sensed by any type of sensor, such as visual light cameras, audio sensors, sensors capable of sensing infrared light, radar imaging, ultrasound, electro-optics, radiography, LIDAR (light detection and ranging), etc. Sensing may include generating samples (e.g., pixels, audio signals) representing signals transmitted or otherwise arriving at the sensor.

[0154] This specification and / or the accompanying drawings may refer to bridging elements. Bridging elements can be implemented in software or hardware. Different bridging elements in a given iteration are configured to apply different mathematical functions to the inputs they receive. Non-limiting examples of mathematical functions include filtering, although other functions may also be applied.

[0155] This specification and / or accompanying drawings may refer to a conceptual structure. A conceptual structure may include one or more clusters. Each cluster may include characteristics and associated metadata. Each reference to one or more clusters may apply to a reference to the conceptual structure.

[0156] This specification and / or the accompanying drawings may refer to a processor. A processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), full-custom integrated circuits, or combinations of such integrated circuits.

[0157] Any combination of any steps of any method shown in the instruction manual and / or accompanying drawings may be provided.

[0158] Any combination of any subject matter that can be claimed.

[0159] Any combination of systems, units, components, processors, and sensors shown in the specification and / or drawings may be provided.

[0160] Any reference to an object can be applied to a pattern. Accordingly, any reference to object detection can be modified as necessary to apply it to pattern detection.

[0161] A context can be a single location / attribute combination at a point in time. A scene is a logically sequential series of events within a causal reference frame. Any references to scenes should be modified as necessary to suit the context.

[0162] The sensing information unit can be sensed by one or more types of sensors. The one or more sensors can belong to the same device or system, or they can belong to different devices within the system.

[0163] A sensing unit can be provided and can receive one or more sensing information units in front of one or more sensors and / or one or more interfaces. The sensing unit can be configured to receive sensing information units from I / O interfaces and / or from sensors. Following the sensing unit can be a combination of various narrow AI agents—also referred to as a narrow AI agent as a whole.

[0164] The sensing information may be processed before reaching the sensing unit, or it may not be processed at all. Any processing can be provided, such as filtering or noise reduction.

[0165] Further embodiments are listed below.

[0166] Example 1. A computer-implemented method for evaluating a classification process includes receiving, at a processing circuit, classification data generated by an enhanced version of a classification process for testing a sensing information unit; evaluating the classification data across the enhanced version by analyzing the distribution of selected classification values ​​of the classification data; determining the compatibility of the classification process with respect to the received classification data based on the evaluation; and issuing a compatibility indication based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the testing sensing information unit.

[0167] Example 2. The method according to Example 1 further includes the allocation of an assignment to trigger another classification process that classifies the test sense information based on the determined compatibility.

[0168] Example 3. The method according to any of Examples 1-2 further includes routing an enhanced version of the test sensing information unit to another classification process.

[0169] Example 4. According to the method of any of Examples 1-3, an enhanced version of the test sensing information unit is generated by using one or more machine learning processes.

[0170] Example 5. The method according to any one of Examples 1-4 further includes applying a classification process to provide classification data.

[0171] Example 6. The method according to any one of Examples 1-5 includes determining that the classification process is capable of classifying the test sensing information unit when the classification value is statistically significant.

[0172] Example 7. The method according to any one of Examples 1-6 includes determining that the classification process is capable of classifying the test sensing information unit when at least a defined percentage of classification values ​​are the same.

[0173] Example 8. The method according to any one of Examples 1-7, wherein the classification process is an embedding-based classification process.

[0174] Example 9. A non-transitory computer-readable medium for evaluating a classification process, the non-transitory computer-readable medium storing instructions that, once executed by a computerized system, cause the computerized system to: receive classification data generated by an enhanced version of a classification process for testing a sensing information unit; evaluate the classification data across the enhanced version by analyzing the distribution of selected classification values ​​of the classification data; determine the compatibility of the classification process with respect to the received classification data based on the evaluation; and issue a compatibility indication based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the testing sensing information unit.

[0175] Example 10. The non-transitory computer-readable medium according to Example 9 also stores instructions for triggering the allocation of another classification process to classify test sense information based on the determined compatibility.

[0176] Example 11. The non-transitory computer-readable medium according to any of the embodiments of Examples 9-10 also stores instructions for routing an enhanced version of the test sensing information unit to another classification process.

[0177] Example 12. A non-transitory computer-readable medium according to any of Examples 9-11, wherein an enhanced version of the test sensing information unit is generated by using one or more machine learning processes.

[0178] Example 13. A non-transitory computer-readable document according to any of Examples 9-12 also stores instructions for applying a classification process to provide classification data.

[0179] Example 14. A non-transitory computer-readable medium according to any of Examples 9-13, including determining that a classification process is capable of classifying test sensing information units when the classification value is statistically significant.

[0180] Example 15. A non-transitory computer-readable medium according to any of Examples 9-14, comprising determining that a classification process is capable of classifying test sensing information units when at least a defined percentage of classification values ​​are the same.

[0181] Example 16. A non-transitory computer-readable medium according to any of Examples 9-15, wherein the classification process is based on an embedded classification process.

[0182] Example 17. A computerized system for evaluating a classification process, the computerized system comprising: a memory unit configured to store classification data generated by an enhanced version of a classification process for testing a sensing information unit; and processing circuitry configured to: evaluate the classification data across the enhanced version by analyzing the distribution of selected classification values ​​of the classification data; determine the compatibility of the classification process with respect to the received classification data based on the evaluation; and issue a compatibility indication based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the testing sensing information unit.

[0183] Example 18. According to the computerized system of Example 17, the processing circuitry is further configured to trigger the allocation of another classification process to classify the test sensing information based on the determined compatibility.

[0184] Example 19. A computerized system according to any of the embodiments of Examples 17-18, wherein the processing circuitry is further configured to route an enhanced version of the test sensing information unit to another classification process.

[0185] Example 20. A computerized system according to any of Examples 17-19, wherein an enhanced version of the test sensing information unit is generated using one or more machine learning processes.

Claims

1. A computer-implemented method for evaluating a classification process, comprising: Receive classification data generated by an enhanced classification process used to test the sensing information unit; The classification data is evaluated across the enhanced version by analyzing the distribution of the selected classification values ​​of the classification data. The compatibility of the classification process with respect to the received classification data is determined based on the assessment. as well as A compatibility indication is issued based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the test sensing information unit.

2. The method of claim 1, further comprising the allocation of an assignment to trigger another classification process that classifies the test sense information based on the determined compatibility.

3. The method of claim 2, further comprising routing the enhanced version of the test sensing information unit to the other classification process.

4. The method of claim 1, wherein the enhanced version of the test sensing information unit is generated by using one or more machine learning processes.

5. The method of claim 1, further comprising applying the classification process to provide the classification data.

6. The method of claim 1, further comprising determining that the classification process is capable of classifying the test sensing information unit when the classification value is statistically significant.

7. The method of claim 1, further comprising determining that the classification process is capable of classifying the test sensing information unit when at least a defined percentage of the classification values ​​are the same.

8. The method of claim 1, wherein the classification process is an embedded classification process.

9. A non-transitory computer-readable medium for evaluating a classification process, the non-transitory computer-readable medium storing instructions that, when executed by a computerized system, cause the computerized system to: Receive classification data generated by an enhanced classification process used to test the sensing information unit; The classification data is evaluated across the enhanced version by analyzing the distribution of the selected classification values ​​of the classification data. The compatibility of the classification process with respect to the received classification data is determined based on the assessment. as well as A compatibility indication is issued based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the test sensing information unit.

10. A computerized system for evaluating a classification process, the computerized system comprising: The memory unit is configured to store classification data generated by an enhanced classification process used to test the sensing information unit; and A processing circuit, communicatively connected to the memory unit, is configured to: The classification data is evaluated across the enhanced version by analyzing the distribution of the selected classification values ​​of the classification data. The compatibility of the classification process with respect to the received classification data is determined based on the assessment. as well as A compatibility indication is issued based on the determined compatibility, the compatibility indication indicating the compatibility of the classification process used to classify elements captured by the test sensing information unit.

Images